DEV Community: Mukit, Ataul

Fix for Crow C++ websocket which could not receive beyond 128 chars

Mukit, Ataul — Fri, 12 Sep 2025 17:12:00 +0000

Context of the post can be found here

Chatgpt modified the websocket.h file and made it workable. Working codes below:

#pragma once
#include 
#include 
#include "crow/socket_adaptors.h"
#include "crow/http_request.h"
#include "crow/TinySHA1.hpp"

namespace crow
{
    namespace websocket
    {
        enum class WebSocketReadState
        {
            MiniHeader,
            Len16,
            Len64,
            Mask,
            Payload,
        };

        struct connection
        {
            virtual void send_binary(const std::string& msg) = 0;
            virtual void send_text(const std::string& msg) = 0;
            virtual void close(const std::string& msg = "quit") = 0;
            virtual ~connection(){}

            void userdata(void* u) { userdata_ = u; }
            void* userdata() { return userdata_; }

        private:
            void* userdata_;
        };

        template 
        class Connection : public connection
        {
            public:
                Connection(const crow::request& req, Adaptor&& adaptor, 
                        std::function open_handler,
                        std::function message_handler,
                        std::function close_handler,
                        std::function error_handler,
                        std::function accept_handler)
                    : adaptor_(std::move(adaptor)), open_handler_(std::move(open_handler)), message_handler_(std::move(message_handler)), close_handler_(std::move(close_handler)), error_handler_(std::move(error_handler))
                    , accept_handler_(std::move(accept_handler))
                {
                    if (!boost::iequals(req.get_header_value("upgrade"), "websocket"))
                    {
                        adaptor.close();
                        delete this;
                        return;
                    }

                    if (accept_handler_)
                    {
                        if (!accept_handler_(req))
                        {
                            adaptor.close();
                            delete this;
                            return;
                        }
                    }

                    // Sec-WebSocket-Key: dGhlIHNhbXBsZSBub25jZQ==
                    // Sec-WebSocket-Version: 13
                    std::string magic = req.get_header_value("Sec-WebSocket-Key") +  "258EAFA5-E914-47DA-95CA-C5AB0DC85B11";
                    sha1::SHA1 s;
                    s.processBytes(magic.data(), magic.size());
                    uint8_t digest[20];
                    s.getDigestBytes(digest);   
                    start(crow::utility::base64encode((char*)digest, 20));
                }

                template
                void dispatch(CompletionHandler handler)
                {
                    adaptor_.get_io_service().dispatch(handler);
                }

                template
                void post(CompletionHandler handler)
                {
                    adaptor_.get_io_service().post(handler);
                }

                void send_pong(const std::string& msg)
                {
                    dispatch([this, msg]{
                        char buf[3] = "\x8A\x00";
                        buf[1] += msg.size();
                        write_buffers_.emplace_back(buf, buf+2);
                        write_buffers_.emplace_back(msg);
                        do_write();
                    });
                }

                void send_binary(const std::string& msg) override
                {
                    dispatch([this, msg]{
                        auto header = build_header(2, msg.size());
                        write_buffers_.emplace_back(std::move(header));
                        write_buffers_.emplace_back(msg);
                        do_write();
                    });
                }

                void send_text(const std::string& msg) override
                {
                    dispatch([this, msg]{
                        auto header = build_header(1, msg.size());
                        write_buffers_.emplace_back(std::move(header));
                        write_buffers_.emplace_back(msg);
                        do_write();
                    });
                }

                void close(const std::string& msg) override
                {
                    dispatch([this, msg]{
                        has_sent_close_ = true;
                        if (has_recv_close_ && !is_close_handler_called_)
                        {
                            is_close_handler_called_ = true;
                            if (close_handler_)
                                close_handler_(*this, msg);
                        }
                        auto header = build_header(0x8, msg.size());
                        write_buffers_.emplace_back(std::move(header));
                        write_buffers_.emplace_back(msg);
                        do_write();
                    });
                }

            protected:

                std::string build_header(int opcode, size_t size)
                {
                    char buf[2+8] = "\x80\x00";
                    buf[0] += opcode;
                    if (size < 126)
                    {
                        buf[1] += size;
                        return {buf, buf+2};
                    }
                    else if (size < 0x10000)
                    {
                        buf[1] += 126;
                        *(uint16_t*)(buf+2) = htons((uint16_t)size);
                        return {buf, buf+4};
                    }
                    else
                    {
                        buf[1] += 127;
                        *(uint64_t*)(buf+2) = ((1==htonl(1)) ? (uint64_t)size : ((uint64_t)htonl((size) & 0xFFFFFFFF) << 32) | htonl((size) >> 32));
                        return {buf, buf+10};
                    }
                }

                void start(std::string&& hello)
                {
                    static std::string header = "HTTP/1.1 101 Switching Protocols\r\n"
                        "Upgrade: websocket\r\n"
                        "Connection: Upgrade\r\n"
                        "Sec-WebSocket-Accept: ";
                    static std::string crlf = "\r\n";
                    write_buffers_.emplace_back(header);
                    write_buffers_.emplace_back(std::move(hello));
                    write_buffers_.emplace_back(crlf);
                    write_buffers_.emplace_back(crlf);
                    do_write();
                    if (open_handler_)
                        open_handler_(*this);
                    do_read();
                }

                void do_read()
                {
                    is_reading = true;
                    switch(state_)
                    {
                        case WebSocketReadState::MiniHeader:
                            {
                                //boost::asio::async_read(adaptor_.socket(), boost::asio::buffer(&mini_header_, 1), 
                                boost::asio::async_read(adaptor_.socket(), boost::asio::buffer(&mini_header_, 2), 
                                    [this](const boost::system::error_code& ec, std::size_t 
#ifdef CROW_ENABLE_DEBUG
                                        bytes_transferred
#endif
                                        )

                                    {
                                        is_reading = false;
                                        mini_header_ = ntohs(mini_header_);
#ifdef CROW_ENABLE_DEBUG
                                        
                                        if (!ec && bytes_transferred != 2)
                                        {
                                            throw std::runtime_error("WebSocket:MiniHeader:async_read fail:asio bug?");
                                        }
#endif

                                        if (!ec && ((mini_header_ & 0x80) == 0x80))
                                        {
                                            if ((mini_header_ & 0x7f) == 127)
                                            {
                                                state_ = WebSocketReadState::Len64;
                                            }
                                            else if ((mini_header_ & 0x7f) == 126)
                                            {
                                                state_ = WebSocketReadState::Len16;
                                            }
                                            else
                                            {
                                                remaining_length_ = mini_header_ & 0x7f;
                                                state_ = WebSocketReadState::Mask;
                                            }
                                            do_read();
                                        }
                                        else
                                        {
                                            close_connection_ = true;
                                            adaptor_.close();
                                            if (error_handler_)
                                                error_handler_(*this);
                                            check_destroy();
                                        }
                                    });
                            }
                            break;
                        case WebSocketReadState::Len16:
                            {
                                boost::asio::async_read(adaptor_.socket(), boost::asio::buffer(&ext_len16_, 2),
                                    [this](const boost::system::error_code& ec, std::size_t 
#ifdef CROW_ENABLE_DEBUG
                                        bytes_transferred
#endif
                                    )
                                    {
                                        is_reading = false;
#ifdef CROW_ENABLE_DEBUG
                                        if (!ec && bytes_transferred != 2)
                                        {
                                            throw std::runtime_error("WebSocket:Len16:async_read fail:asio bug?");
                                        }
#endif
                                        if (!ec)
                                        {
                                            remaining_length_ = ntohs(ext_len16_);
                                            state_ = WebSocketReadState::Mask;
                                            do_read();
                                        }
                                        else
                                        {
                                            close_connection_ = true;
                                            adaptor_.close();
                                            if (error_handler_)
                                                error_handler_(*this);
                                            check_destroy();
                                        }
                                    });
                            }
                            break;
                        case WebSocketReadState::Len64:
                            {
                                boost::asio::async_read(adaptor_.socket(), boost::asio::buffer(&ext_len64_, 8),
                                    [this](const boost::system::error_code& ec, std::size_t 
#ifdef CROW_ENABLE_DEBUG
                                        bytes_transferred
#endif
                                    )
                                    {
                                        is_reading = false;
#ifdef CROW_ENABLE_DEBUG
                                        if (!ec && bytes_transferred != 8)
                                        {
                                            throw std::runtime_error("WebSocket:Len64:async_read fail:asio bug?");
                                        }
#endif
                                        if (!ec)
                                        {
                                            uint64_t v = ext_len64_;
                                            if (htonl(1) != 1) {
                                                v = ( (uint64_t)ntohl((uint32_t)(ext_len64_ & 0xFFFFFFFF)) << 32 )
                                                  | (uint64_t)ntohl((uint32_t)(ext_len64_ >> 32));
                                            }
                                            remaining_length_ = v;
                                            state_ = WebSocketReadState::Mask;
                                            do_read();
                                        }
                                        else
                                        {
                                            close_connection_ = true;
                                            adaptor_.close();
                                            if (error_handler_)
                                                error_handler_(*this);
                                            check_destroy();
                                        }
                                    });
                            }
                            break;
                        case WebSocketReadState::Mask:
                                boost::asio::async_read(adaptor_.socket(), boost::asio::buffer((char*)&mask_, 4), 
                                    [this](const boost::system::error_code& ec, std::size_t 
#ifdef CROW_ENABLE_DEBUG 
                                        bytes_transferred
#endif
                                    )
                                    {
                                        is_reading = false;
#ifdef CROW_ENABLE_DEBUG
                                        if (!ec && bytes_transferred != 4)
                                        {
                                            throw std::runtime_error("WebSocket:Mask:async_read fail:asio bug?");
                                        }
#endif

                                        if (!ec)
                                        {
                                            state_ = WebSocketReadState::Payload;
                                            do_read();
                                        }
                                        else
                                        {
                                            close_connection_ = true;
                                            if (error_handler_)
                                                error_handler_(*this);
                                            adaptor_.close();
                                        }
                                    });
                            break;
                        case WebSocketReadState::Payload:
                            {
                                size_t to_read = buffer_.size();
                                if (remaining_length_ < to_read)
                                    to_read = remaining_length_;
                                adaptor_.socket().async_read_some( boost::asio::buffer(buffer_, to_read), 
                                    [this](const boost::system::error_code& ec, std::size_t bytes_transferred)
                                    {
                                        is_reading = false;

                                        if (!ec)
                                        {
                                            fragment_.insert(fragment_.end(), buffer_.begin(), buffer_.begin() + bytes_transferred);
                                            remaining_length_ -= bytes_transferred;
                                            if (remaining_length_ == 0)
                                            {
                                                handle_fragment();
                                                state_ = WebSocketReadState::MiniHeader;
                                                do_read();
                                            }
                                        }
                                        else
                                        {
                                            close_connection_ = true;
                                            if (error_handler_)
                                                error_handler_(*this);
                                            adaptor_.close();
                                        }
                                    });
                            }
                            break;
                    }
                }

                bool is_FIN()
                {
                    return mini_header_ & 0x8000;
                }

                int opcode()
                {
                    return (mini_header_ & 0x0f00) >> 8;
                }

                void handle_fragment()
                {
                    for(decltype(fragment_.length()) i = 0; i < fragment_.length(); i ++)
                    {
                        fragment_[i] ^= ((char*)&mask_)[i%4];
                    }
                    switch(opcode())
                    {
                        case 0: // Continuation
                            {
                                message_ += fragment_;
                                if (is_FIN())
                                {
                                    if (message_handler_)
                                        message_handler_(*this, message_, is_binary_);
                                    message_.clear();
                                }
                            }
                        case 1: // Text
                            {
                                is_binary_ = false;
                                message_ += fragment_;
                                if (is_FIN())
                                {
                                    if (message_handler_)
                                        message_handler_(*this, message_, is_binary_);
                                    message_.clear();
                                }
                            }
                            break;
                        case 2: // Binary
                            {
                                is_binary_ = true;
                                message_ += fragment_;
                                if (is_FIN())
                                {
                                    if (message_handler_)
                                        message_handler_(*this, message_, is_binary_);
                                    message_.clear();
                                }
                            }
                            break;
                        case 0x8: // Close
                            {
                                has_recv_close_ = true;
                                if (!has_sent_close_)
                                {
                                    close(fragment_);
                                }
                                else
                                {
                                    adaptor_.close();
                                    close_connection_ = true;
                                    if (!is_close_handler_called_)
                                    {
                                        if (close_handler_)
                                            close_handler_(*this, fragment_);
                                        is_close_handler_called_ = true;
                                    }
                                    check_destroy();
                                }
                            }
                            break;
                        case 0x9: // Ping
                            {
                                send_pong(fragment_);
                            }
                            break;
                        case 0xA: // Pong
                            {
                                pong_received_ = true;
                            }
                            break;
                    }

                    fragment_.clear();
                }

                void do_write()
                {
                    if (sending_buffers_.empty())
                    {
                        sending_buffers_.swap(write_buffers_);
                        std::vector buffers;
                        buffers.reserve(sending_buffers_.size());
                        for(auto& s:sending_buffers_)
                        {
                            buffers.emplace_back(boost::asio::buffer(s));
                        }
                        boost::asio::async_write(adaptor_.socket(), buffers, 
                            [&](const boost::system::error_code& ec, std::size_t /*bytes_transferred*/)
                            {
                                sending_buffers_.clear();
                                if (!ec && !close_connection_)
                                {
                                    if (!write_buffers_.empty())
                                        do_write();
                                    if (has_sent_close_)
                                        close_connection_ = true;
                                }
                                else
                                {
                                    close_connection_ = true;
                                    check_destroy();
                                }
                            });
                    }
                }

                void check_destroy()
                {
                    //if (has_sent_close_ && has_recv_close_)
                    if (!is_close_handler_called_)
                        if (close_handler_)
                            close_handler_(*this, "uncleanly");
                    if (sending_buffers_.empty() && !is_reading)
                        delete this;
                }
            private:
                Adaptor adaptor_;

                std::vector sending_buffers_;
                std::vector write_buffers_;

                boost::array buffer_;
                bool is_binary_;
                std::string message_;
                std::string fragment_;
                WebSocketReadState state_{WebSocketReadState::MiniHeader};
                uint64_t remaining_length_{0};
                bool close_connection_{false};
                bool is_reading{false};
                uint32_t mask_;
                uint16_t mini_header_;
                
                uint16_t ext_len16_{0};
                uint64_t ext_len64_{0};
bool has_sent_close_{false};
                bool has_recv_close_{false};
                bool error_occured_{false};
                bool pong_received_{false};
                bool is_close_handler_called_{false};

                std::function open_handler_;
                std::function message_handler_;
                std::function close_handler_;
                std::function error_handler_;
                std::function accept_handler_;
        };
    }
}

Business Rules Conversion Prompt

Mukit, Ataul — Fri, 15 Aug 2025 07:09:00 +0000

Excerpt from Ray Garcia:

This is an extensive version of a business rules conversion prompt. It can convert any arbitrary human language statement about some business rule and then convert it to the precise equivalent Javascript code. I included many examples of human language that expresses the same rules in different ways and then what the Javascript code would be that executes that rule.

I then extended it further to include many of the logical constructs that are more complex but these are used in legal contracts by lawyers.

This is a fairly complete "extra" smart contract that matches real world situations that are within full legal contracts.

I tested it with a small model, Mistral 7B instruct and it works with that model.

I used large model to generate a lot of these examples, Anthropic Opus which is the most advanced model they have.

What I did afterwards to test it is generate a set of rules for the insurance industry, which is very complex, and then convert it to Javascript code. Since I only did it based on the rules, it lacked a schema, so I did the reverse and generated a JSON schema to support the rules, and then from that I generated code to implement that schema on RocksDB key-value database.

This is interesting in that it reverses the entire process, where instead of trying to figure out the data structure it started with human language describing some rules and then from that it generated the structure to support the data. This closer to how people think.

What is a leap here is that it might be possible to take some legal document, with an LLM parse it into the constituent rules and then implement it as a verifiable software that maintains the rules of the contract.

I have spent a lot of time creating specialized scripting languages to control an LLM to do this but this is the first time that it get beyond my own experiments and into something that could be implemented.

Why this is such a breakthrough is that LLM's are non-deterministic so even though I have several scripting languages I use for prompting them, they remain unreliable. Converting it to code makes it more reliable.

What was missing is a stack that is lean and would scale. Doing this with python and postgres on fat hosting isn't practical, it becomes a mess and very costly on the big tech hosting and then it won't scale due to many factors like code bloat.

By using the LLM as the mediator that bridge between human language and the code it can create solutions fast that are lean if it has a good stack to run on.

Example Doc

Install Quarks in Ubuntu 20.04 and above

Mukit, Ataul — Wed, 19 Mar 2025 11:54:41 +0000

Installing and Running Quarks Database on Ubuntu

Quarks is a lightweight, high-performance database designed for specific workloads. This tutorial will guide you through installing and running Quarks on an Ubuntu system.

Prerequisites

Before installing Quarks, ensure that your system is up to date. Run the following command:

sudo apt-get update -y

Step 1: Install Required Dependencies

Quarks requires several essential dependencies to be installed before proceeding with the build.

sudo apt install cmake  
sudo apt-get install build-essential  
sudo apt-get install ninja-build

These packages include essential build tools like CMake, Ninja, and build-essential.

Step 2: Install Boost 1.69

Quarks depends on Boost libraries, so we need to install Boost 1.69 manually.

Download Boost 1.69

Navigate to /usr/local/src and download the Boost 1.69 source code using wget:

cd /usr/local/src  
sudo wget -O boost_1_69_0.tar.gz https://archives.boost.io/release/1.69.0/source/boost_1_69_0.tar.gz

Extract the Archive

Unpack the downloaded archive:

sudo tar -xvzf boost_1_69_0.tar.gz  
cd boost_1_69_0

Build and Install Boost

Run the following commands to bootstrap and install Boost:

sudo ./bootstrap.sh --prefix=/usr/local  
sudo ./b2 install

This process will compile and install Boost in /usr/local.

Step 3: Install Additional Dependencies

Install other required dependencies:

sudo apt-get install libv8-dev  # (Optional: Only needed if using the V8 engine)  
sudo apt-get install librocksdb-dev  
sudo apt-get install libzmq3-dev

These libraries are required for Quarks' functionality.

Step 4: Clone and Build Quarks

Clone the Repository

Retrieve the Quarks source code from GitHub:

git clone https://github.com/lucpattyn/quarks.git  
cd quarks

Build Quarks

Create a build directory and use CMake with Ninja to compile Quarks:

mkdir build  
cd build  
cmake .. -G Ninja  
ninja

Step 5: Run Quarks

Once the build is complete, you can run Quarks using the following command:

./ocv_microservice_crow

This will start the Quarks microservice.

Conclusion

By following this guide, you have successfully installed and run Quarks on your Ubuntu system. If you encounter any issues, ensure that all dependencies are correctly installed and try re-running the build steps.

Install Boost in Ubuntu

Mukit, Ataul — Wed, 19 Mar 2025 11:48:38 +0000

Go to directory and Use wget to download the Boost 1.69 source code:

cd /usr/local/src

sudo wget -O boost_1_69_0.tar.gz 

https://archives.boost.io/release/1.69.0/source/boost_1_69_0.tar.gz

To verify the integrity of the downloaded file, check its size:

ls -lh boost_1_69_0.tar.gz

The correct file size should be around 116 MB.

Extract the Archive

sudo tar -xvzf boost_1_69_0.tar.gz

cd boost_1_69_0/

Boost's bootstrap setup:

./bootstrap.sh --prefix=/usr/

Then build it with:

./b2

Connecting LLM to a Real-World Robot

Mukit, Ataul — Sun, 17 Nov 2024 16:18:39 +0000

Connecting a Real-World Robot to an AI-Powered Language Model

In today’s rapidly evolving technological landscape, connecting a real-world robot to an advanced AI language model offers an exciting opportunity for intuitive human-machine interaction. Imagine giving a natural language command like “Clean the room and avoid obstacles,” and having a robot understand and execute it seamlessly. Here's how to make this vision a reality.

1. Define Your Robot’s Capabilities

Start by clearly defining what your robot can do. This step lays the foundation for translating user commands into actionable tasks. Key functions might include:

Movement: Forward, backward, turning, or stopping.
Task Execution: Picking up objects, cleaning, scanning, or performing other specialized functions.
Sensors: Using environmental sensors for navigation and obstacle detection.

2. Establish a Communication Interface

To enable communication between the language model and the robot, you need a robust interface. Here are some common methods:

APIs: If the robot provides an API, HTTP requests can be used to send commands.
Network Sockets: Real-time communication via WebSockets or MQTT.
Serial Communication: Using USB or Bluetooth to send commands directly through serial protocols.

This interface ensures a smooth relay of instructions from the AI model to the robot.

3. Map Commands to Robot Actions

A middleware layer is essential to bridge natural language input and machine-understandable commands. For instance:

Natural Language Input: “Move forward for 5 seconds.”
Mapped Command: robot.move('forward', 5)

This can be implemented using a programming language like Python or JavaScript, where scripts interpret AI-generated text into specific robot instructions.

4. Enable Real-Time Connection

To process user commands effectively, integrate the language model into a real-time system:

Web Interface: Host the AI on a local or cloud-based server, allowing users to input commands via a chat or voice interface.
Backend Logic: Process AI responses in the backend and convert them into executable robot commands.

5. Integrate Sensors for Feedback

Robots often rely on sensors to interact with their environment. Incorporating feedback loops ensures real-time awareness and adaptive behavior. For example:

The robot sends confirmation upon completing a task.
Sensors detect obstacles and relay data to adjust actions accordingly.

This integration enhances the robot's ability to respond dynamically to its environment.

6. Step-by-Step Workflow

Example Scenario:

User Command: “Clean the room and avoid obstacles.”
AI Interpretation: The AI translates this into actionable steps: "Move forward, scan for obstacles, turn as needed."
Command Execution: The backend converts the instruction to a robot-specific command: robot.start_cleaning().
Robot Action: The robot initiates cleaning, adjusting movements based on sensor data.

7. Tools and Frameworks

Several tools can streamline this process:

Python Libraries: pyserial, mqtt, or websockets for communication.
ROS (Robot Operating System): A flexible framework for interfacing with robots.
Node.js: Use socket.io or node-serialport for real-time control.

8. Ensure Safety and Error Handling

Robots must prioritize safety. Implement constraints to avoid unsafe commands, such as preventing collisions or excessive speeds. Robust error handling can notify users of failed commands or system malfunctions, ensuring reliability.

9. Expanding Capabilities

For a more advanced system, consider:

Speech-to-Text (STT): Allowing voice commands for seamless interaction.
Adaptive Learning: Training the robot with reinforcement learning for improved decision-making.

Conclusion

By integrating a real-world robot with an AI-powered language model, you can create a system that understands natural language and performs tasks with remarkable precision. This synergy of AI and robotics represents the future of intuitive automation, where machines can respond intelligently to human needs.

Story of a failed entrepreneur

Mukit, Ataul — Thu, 14 Nov 2024 13:00:05 +0000

Chapter 1: IF YOU ARE NOT IN - YOU ARE OUT!

Entrepreneurship is Ownership

It was a cold(!) July morning in a hot summer that I learnt one of the first hard lessons about entrepreneurship : if you’re not deeply involved, fully invested, and genuinely present (financially) in your business, it won’t hesitate to spit (read screw) you out. Entrepreneurship is unforgiving to passivity. Just being around is not enough. You have to be all in, in every aspect.

Elon Musk once said, "If you’re co-founder or CEO, you have to do all kinds of tasks you might not want to do... If you don’t do your chores, the company won’t succeed. No task is too menial." That couldn’t be more accurate. Successful businesses demand everything from you. The moment you check out or look for an easy way, you start inching toward the exit.

Looking to other entrepreneurs was valuable, but not because I could follow their exact path. They inspired me to create my own. Each decision had to be mine, each challenge personally handled. Ownership isn’t just having a title; it’s being truly, completely involved in the vision, the work, and the grind. Anything less, and you’re out.

The moment I stepped into entrepreneurship, it was like diving into the deep end of a pool—sink or swim. I learned fast that there’s no free ride, no shortcuts through someone else’s success. I had to own every decision, every failure, and every late night, knowing that no one could carry me across the finish line but me.

As Reid Hoffman, founder of LinkedIn, put it, "An entrepreneur is someone who will jump off a cliff and assemble an airplane on the way down." That’s the game I signed up for—taking the plunge and piecing things together midair. Looking up to successful people became less about finding a map and more about gathering inspiration to make my own.

Do not count on getting a free ride anywhere, and do not count on riding on other people’s success. You have to do what you have to do on your own !

Chapter 2: WHEN THINGS MOVE!

When you move things start moving

I didn’t expect progress to feel so repetitive. Often, I was just going through the motions, uncertain if any of it would pay off. But I realized that even the smallest actions can spark momentum. “The way to get started is to quit talking and begin doing,” said Walt Disney, and that became my guiding principle. Just moving, no matter how small, made things around me start to shift.

In tough times, just staying in motion felt like a victory. I learned that success doesn’t always arrive with a bang; sometimes it sneaks in quietly, the product of small, almost invisible efforts piling up.

Chapter 3: INVESTMENT AND FEAR

Fear of fear is the greatest fear

Money is the worst investment if it’s your only investment in business.
Like many, I thought funding would solve everything. But when the funds were tight, I had to dig deeper, investing time, grit, and my mental energy. And when fear crept in, I discovered something else: fear itself wasn’t the enemy. It was my hesitation to face it. Fear became a test of resilience.

"Fear is the disease," said tech entrepreneur and philanthropist Naveen Jain, "hustle is the antidote." There’s no way around fear except through it. The lesson wasn’t about amassing cash but amassing courage and the strength to hustle through dark days. That became the real investment.

Chapter 4: THE PRODUCT!

It was never about the product but the idea of the product

In the beginning, I was focused on the product itself, convinced that I could perfect it. I learned the hard way that it’s not the product but the idea of the product that counts. Technology is only as powerful as the clarity of the idea behind it. Steve Jobs put it best when he said, "Innovation is saying no to a thousand things." This wasn’t just about making something shiny and new—it was about refining my vision.

A true partnership, I realized, shouldn’t feel like a circle where everyone is chasing their own tail but like a vector, where each person brings momentum, aiming in a fresh direction. And, most importantly, when a problem arises, owning it is the only way to solve it. Without commitment to the idea, you’re stuck. Innovation is about moving past that fear, about not letting your product trap you in yesterday’s successes.

Chapter 5: THE FINE LINE

You think you are doing a good job, but that was not your job

There’s a fine line between being an entrepreneur and simply being lazy. I learned that real growth comes not from repeating what I’m good at but from embracing what I’m not.

This line between innovation and inertia is thin, and crossing it takes vigilance. Jeff Bezos once said, "I knew that if I failed I wouldn’t regret that, but I knew the one thing I might regret is not trying." The antidote to compromise is honest effort; without it, you fall into the trap of mental laziness, giving up bits of your vision piece by piece. I realized that if I wanted a team with ambition and edge, I’d have to learn to manage the rattlesnakes—the aggressive, hungry people—who’d push me as hard as I pushed them. As destiny would have had it, I was pushed outside - the circle and my own creations.

Chapter 6: RISE ABOVE HATE

John Cena anyone?

There came a point where resentment started to creep in judgment. Jealousy without ambition—that’s the kind of thing that can wreck an entire business from the inside as well as the self.

But I realized something powerful: doing something constructive is the only way to rise above resentment. Instead of blaming others, I had to learn to accept responsibility. As Warren Buffett said, "You only have to do a very few things right in your life so long as you don’t do too many things wrong."

Chapter 7: LIFE OVERRATES TO BERATE

*Maybe I will find it .. *

At 25, I believed in the extraordinary. I thought life would hand me success if I worked hard enough. Ten years later, I learned the bitter truth: you have to make it happen. As the years passed, that extraordinary belief got tested, torn apart by reality. Only the true believers or the damned keep going after those first years, after failure upon failure.

Steve Jobs said, "The only way to do great work is to love what you do." Passion is important, yes, but it isn’t enough. You have to be willing to take hits, to look stupid, to get back up even when it feels like the world is watching you stumble.

Life doesn’t give us a clean slate. I felt it was a bit overrated and not too much to extract from it as years when by. I failed in a lot of way, not because I was unlucky, but because I never stopped trying. I don’t have a clear endpoint in mind, just a sense that there’s more. Maybe I’ll find it. Maybe I won’t. But I’ll keep searching because that’s the only way to be truly alive, even if the world tells me otherwise.

So, Compadres, let’s not give up hope! If the search brings us moments of brilliance, like sparks from the first fire, let’s keep going.

Role based UI Access

Mukit, Ataul — Tue, 29 Oct 2024 13:38:36 +0000

Complexities of Role based UI Access

In some projects I came across that implemented roles (for example, super admin, admin, regular user etc.), I found a lot of codes based on if-else conditions making the readability complex due to ever-changing requirements of a project. The problem gets more complex whenever there is a case of mixed access, that is multiple roles can access multiple pages both common and exclusive to one another:

Let's say in a hospital management system we have doctors accessing certain pages (ex. Patient History, Generate Prescription, View Report), lab technicians some pages which doctors do not need access and some which they do (Upload Report, View Report) and patients who have access to certain pages(Patient History, View Report, View Prescription) , Admin for the sake of this article all pages of a certain Hospital and Super Admin - all pages for all hospitals.

While reviewing codes for these kind of role implementation, I have seen stuff written with if/else logics to prevent access to the page in a router/controller.

Pseudo codes for illustration purposes):

If role != 'admin' or role != 'doctor'
    return 
    {'code':401, 
    'error':'generate prescription is inaccessible'} ;

And then as project requirements grows the conditions in the if/else start growing and becomes not manageable.

So here is a suggestion which might be worth considering.

Implement a role table. Here table can be considered a matrix with dynamically expandable rows and columns where columns represent the roles and the rows represent the UI pages/controls. Illustration below:

Role vs UI Table

Note that it probably would not be a good idea to use a Relational Database Table although it is looking like one above, simply because the columns are not static.

This is more of a matrix implementation than anything else.

However, the key here is we need an elegant lookup table and the if/else codes mentioned aboive can be changed in a generic fashion likewise:

// assume Role = 'doctor'
// assmue Page = 'generate prescription'

If Accessible(Role, Page) == 'no'
 return 
    {'code':401, 
    'error': Page + ' is inaccessible'}

Note that no matter how may roles or pages you create, the code in router/controller remains intact and need not be touched.

All you have to do is update the Role vs UI table.

There is one thing left and that is - what if a user such as super admin has access to multiple hospitals page and an admin only has access to pages of a certain hospital. Or while a doctor is tied to a certain hospital, a patient can access pages in multiple hospitals that he/she has visited.

We can simply solve it by a "Belongs To" table. Here is the concept in a nutshell:

(User ID | Role) vs Belongs To Table

[user1 | admin, hospital 1]

[user2 | admin, hospital 2]

[user3 | doctor, hospital 1]

[user4 | doctor, hospital 2]

[user5 | labtech, hospital 1]

[user6 | labtech, hospital 2]

[user7 | patient, Any]

[user8 | super, Any]

Here we are simply having a lookup table of combined key using user and role vs which hospital that user belongs to based on that role.

Super admin and patient can belong to any hospital.

Therefore, before applying the Accessible method, we just do a simple check with Belongs To, i.e

hospital = BelongsTo(userId, Role) 
If hospital == 'any' or hospital == currentHospital
   then return check_accessibility();

Whatever discussed in this article is a guideline to the beginners (or sometimes mid level devs) to think in a more organized way to deal with roles and accessibility. Definitely a real world project demands more but we have to have the right mindset first and start writing elegant codes.

Critical Cases to Handle in a Real-Time Chat Application

Mukit, Ataul — Mon, 16 Jan 2023 18:06:00 +0000

Apart from the known challenges of scaling the backend of a chatting system or service, there are few problems that needs to be tackled to give users a smooth experience.
From the experience of implementing a scalable chat system, my team encountered few unusual circumstances that we didn't anticipate. Some of them listed below:

One of the biggest challenges is detecting a closed socket. Sometimes when an app (somehow browsers are more well behaved) exits (read killed), it fails to notify the chat server that the client has died; as a result - the server still thinks the socket socket is alive and hence you might find the user associated with the socket still showing online. One way to handle the situation would be to run a cron an identify which users has been inactive for some time. Send them a ping from the server and if the message sending fails, then remove them from online list. The other way which probably needs more hit handling capabilities in server is send a ping at certain intervals like 30 seconds and if a user hasn't sent ping in more than a 40 seconds, mark them as offline. Solutions can be multiple, but what's important is to know that sockets may not behave as expected when it comes to detecting disconnection.
The nightmare for any chatting system is slow connections which frequently drops packets. For starters, even if you start with some wonderful package like socket.io (probably that's the simplest to start with), you have to keep in mind that in future you may have to switch to raw websockets/own protocols or probably webrtc even for low latency solution. You may not go for XMPP protocol (which is pretty much the standard) considering resource shortages and learning curve; but as the project matures, it is possible to bring in other protocols when you gain further knowledge about the whole ecosystem and how to abstract and optimize.
One difficult challenge is to ensure user is properly getting unread messages in an "on and off " constantly dropping internet connection. If a user is offline, then comes back at later stage, he/she has to be provided the missed messages that were sent when user was offline. Usually that is done in a response to api call or through a socket when user reconnects. The issue that may arise is user may receive same message multiple times through live socket and then again from the API call after reconnecting to server - since reconnection may occur frequently in an "on and off" connection. The real challenge is in the client side because usually the message coming through live socket arrives in a different thread than the one which is a response to an API call. Hence the user ends up getting duplicate messages and duplicate checking in multiple threads can pose a challenge if not handed well.
A critical challenge is to update the status of messages i.e pending, delivered, seen etc. properly in the client end. A user can log in, see a message then disappear (read go offline) and the other party may log in at a later time to see the message status whether it was delivered to or seen by the disappeared user. Doing it in a an optimised way requires a lot of thinking. Even implementing XMPP has it's own challenges because it provides a subset of message status range by default and someone needs to know the protocol ins and out to tweak it.
We have just gone through few of the major challenges, that too in an 1-1 chatting system. Doing it for group chatting has its own challenges. However, if the 4 points mentioned above is handled well for a chatting system, the challenges posted by group chat implementation becomes a bit easier. So tackling these 4 well is a good start for any chatting system/service.

1-1 WebRTC call with mobile devices

Mukit, Ataul — Wed, 31 Aug 2022 04:50:53 +0000

Although there are many examples related to WebRTC based audio/video calls, specially browser based ones floating in the world wide net, but it is not always easy to conceptualize and implement the concepts when it comes to mobile devices.
Here is a step by step guide (mainly consisting the important steps) to follow through for a stable peer to peer call connecting mechanism for mobiles. This is a very high level diagram, and whoever following it has to have a basic idea of how WebRTC calls work and the basic terms like SDP offer and answer.

Quarks Replication for a scalable solution

Mukit, Ataul — Tue, 26 Apr 2022 11:54:23 +0000

This article is a follow up of the article 'what quarks is about':
Quarks - a new mindset ... (https://dev.to/lucpattyn/quarks-a-new-approach-with-a-new-mindset-to-programming-10lk).
There was another article about architecture, but over time the concept has been simplified, hence the new writeup.

Just to recap, Quarks is a fast information write and querying system and so far we have implemented a single isolated individual node. Now it is time for data replication and making multiple nodes work together for a scalable solution.

It would be easier to explain the whole system with a diagram. The diagram discusses 3 stages one after one for data saving and replication:

First and foremost we need a scalable system to save the data when there are frequent hits, which we are denoting as stage 1.

At this stage, we are using Quarks Writer nodes (we can use as many depending on the traffic) to save data.
Each node creates multiple rocksdb instances in bucket concept because later when we retrieve data from these buckets, we don't have to go over a lot of data. Once all the data of one bucket has been retrieved and replicated, we will delete the buckets, so not a lot of disk space is used. The diagram shows only one bucket per writer node for keeping the illustration clean, but actually each node will create multiple buckets. Once data saving starts, it is time for stage 2 to fetch data and replicate.

In stage 2, we run a cron job to control a Quarks Consumer node to go through all the available writer nodes in round robin fashion and fetch data in bulks. The consumer gets a bulk of data from node 1 (writer node of-course), sends it to a broker which serialises and maintain a message queue (message object is the data + instructions/hints of what to do with the data) to send to required destinations (the primary destination being the message broker, secondary being logging servers and third parties). Then the consumer picks node 2, performs same action and moves on to node 3 and so on. As data starts filling the message queue in broker node, it is time for stage 3.

In this stage, the broker picks messages from the queue, extracts data, sends to Reader Nodes which basically responds back to clients the data when requested. Quarks supports in memory data store, so if required can fetch and send date to clients really fast.

One thing to note is a single Writer Node don't have all the data. Multiple such nodes together holds the whole data.
The message broker is responsible for aggregating data and ensure individual reader nodes have all the data (since there cannot be inconsistency in data when queried).

Quarks is sharding-enabled by nature. You would be able to write logics in the broker (not implemented yet) to select the right readers for buckets of data based on sharding logic.

To re-iterate, simplicity is at the heart of Quarks, and it aims to build a scalable solutions for modern world software development in server side.

Wordcount for huge data

Mukit, Ataul — Thu, 27 May 2021 11:35:48 +0000

If we are to process unique word counts for input data (read many sentences), the solution is pretty straightforward using a map. Javascript example below:

However, if the number of unique words is huge and there are memory limitations, then finding an optimal solution becomes difficult. In that case we can go with a tree traversal route to save memory. Say, we have a to find the count of few words from input data. Words are "Stack", "Stick", "Stuck", "The", "Then". We can form a multi node tree where the leaf nodes will contain the count. Illustration below:

As you can see, instead of having stuck, stack, stick, stuck as separate elements in the map, we are creating a tree to save some letters (bytes) being reused.

Disclaimers note: I don't have the exact algorithms in my head as code yet :) and the representation of "the" and "then" in the image is not exactly right.

Thanks Joynal (https://www.linkedin.com/in/joynaluu/) for bringing up the topic to ponder upon in social media.

signing off,
Mukit

What I think as True Artificial Intelligence

Mukit, Ataul — Wed, 19 May 2021 06:09:52 +0000

When we talk about Artificial Intelligence now a days, for me - it seems pretty dumb intelligence. True intelligence will have the power of inference. For example, from a given training dataset presently the AI may detect a tree (see image below):

However, it can infer from it's knowledge-base the tree can have more branches (this is knowledge, not trained data), can be without leaves and it can also relate to fruits and the fact that fruit grows on trees. Hence it would be possible for the AI to detect in future - a tree without leaves and only branches, or without branches and only leaves, given no prior training data, but simply from the power of inference. This for me is intelligence.