DEV Community: bhargavirengarajan21

Cloud Computing Project Idea

bhargavirengarajan21 — Wed, 25 Feb 2026 18:08:07 +0000

Block chain in Real time cloud logging

bhargavirengarajan21 ・ Mar 24 '25

#web3 #cloudcomputing #kubernetes #docker

Block chain in Real time cloud logging

bhargavirengarajan21 — Mon, 24 Mar 2025 06:19:12 +0000

Can Blockchain and Cloud Work Together for Real-Time Logging?

The number of likes gonna decide my project idea, on an interesting note about the cloud logging system, I and my teammate proposed a real-time logging using Hyperledger

The idea was simple: How can we make cloud logging real-time and secure without compromising performance or cost?

So we asked ourselves, "What if we could combine the speed of real-time processing with the security of blockchain?"

Architecture & Workflow:

Working:

Log Generation: Cloud server generates logs → Sent to Pub/Sub for processing.
Log Routing: Pub/Sub sends logs to Redis for caching and fast access.
Pub/Sub also sends logs for classification (critical vs non-critical).
Critical Log Handling: If it’s a critical log → Log is hashed and sent to Hyperledger for tamper-proof storage. Redis stores the log for quick future access.
Non-Critical Log Handling: If it’s a non-critical log → Sent to traditional logging for storage. Redis stores frequently accessed logs.
Retrieval: If a log is requested → Redis provides a fast response if the log is available. If Redis misses → Hyperledger provides the log → Redis updates its cache.
Monitoring:
Grafana monitors: Redis (real-time performance), Hyperledger (security and consistency), Traditional logging (success/failure feedback)
Failure Handling:
1. If Redis fails → Pub/Sub retries the request or falls back to Hyperledger.
2. If Hyperledger write fails → Log is stored in Redis as "pending" and retried later.
3. If traditional logging fails → Grafana generates an alert.

Advantages:

1. Speed and Real-Time Performance:
Redis + Pub/Sub = High-speed log processing (low latency).
Logs are available instantly for search and monitoring.

2. Security and Data Integrity:
Hyperledger ensures logs are tamper-proof and cryptographically secured.
Blockchain-based hashing guarantees that logs cannot be altered post-storage.

3. Fault Tolerance and High Availability:
Pub/Sub allows retry and message buffering → Prevents data loss.
Redis replication and Hyperledger peer-based consensus ensure high availability.

4. Scalability:
Redis + Pub/Sub = Horizontal scaling for high-volume log handling.
Hyperledger can scale for increased load by adding more peers.
Decoupling of Processing and Storage:
Redis (for speed) and Hyperledger (for integrity) work independently → No bottlenecks.

5. Unified Monitoring:
Grafana provides centralized monitoring across all components (Redis, Hyperledger, traditional logs).
Alerts and health checks ensure system health and proactive failure handling.

6. Compliance and Regulatory Value:
Financial and healthcare industries require immutable logs for audits → Blockchain provides proof of integrity.
Perfect for industries needing regulatory-compliant logging.

Commercial Value:

1. Target Market:

Cloud-native applications (SaaS, web apps, microservices).

Kubernetes deployments needing scalable log pipelines.

DevOps teams seeking real-time insights + security.

2. Competitive Edge:

Datadog and Splunk provide real-time logging but lack blockchain-based integrity. Your solution offers both speed and verifiability → Strong selling point. Multi-cloud support (via Kubernetes) = Portable across AWS, GCP, Azure → Reduces vendor lock-in.

Drawbacks:

1. High Complexity:
More components (Redis + Pub/Sub + Hyperledger) = More maintenance and complexity.
Datadog and Splunk are easier to set up — Your system requires skilled DevOps knowledge.

2. Blockchain Latency:
Hyperledger consensus adds latency (~100ms–500ms) → Could affect real-time responsiveness.
Solution → Cache critical logs in Redis before blockchain commit.

3. Cost of Hyperledger Storage:
Storing full logs in Hyperledger could get expensive at scale.
Solution → Store only log hashes in Hyperledger and full logs in Redis or traditional storage.

4. Limited Hyperledger Scalability:
Hyperledger scales linearly → Adding more peers increases consensus time.
Solution → Use a batching model to reduce the number of consensus events.

5. Pub/Sub Complexity:
Pub/Sub introduces message handling complexity (ordering, retries).
Solution → Use dead-letter queues (DLQ)

Poll & Feedbacks

Do you think its an overkill project?
Can you please explain to me if this a complex and does not solve any purpose?
if its good points to improve ?

Docker storage commands?

bhargavirengarajan21 — Wed, 17 Aug 2022 05:22:57 +0000

Create volume:

$sudo docker volume create  volume-b1
volume-b1

Inspect a Volume

$ sudo docker inspect volume-b1
[
    {
        "CreatedAt": "2022-08-17T09:52:47+05:30",
        "Driver": "local",
        "Labels": {},
        "Mountpoint": "/var/lib/docker/volumes/volume-b1/_data",
        "Name": "volume-b1",
        "Options": {},
        "Scope": "local"
    }
]

list a volume:

$sudo docker volume ls

DRIVER    VOLUME NAME
local     vol-busybox
local     volume-b1

delete a volume:

 $sudo docker volume rm volume-b1

Mount the driver to a container

-v or --volume : contains all fields and also order should be maintained
- name of the volume
- path
- options like ro--(read-only)

$ sudo docker run -d --volume vol-ubuntu:/tmp ubuntu
14efcc03cc75c98877f1074bc30d3570b4f062c122cccb83272409d677c9ae4c

--mount: Separate fields , and contains as key-value pair, order is not important, easier to understand.

$ docker service create \
    --mount 'type=volume/bind/tempfs,src=<VOLUME-NAME>,dst=<CONTAINER-PATH>,volume-driver=local,volume-opt=type=nfs,volume-opt=device=<nfs-server>:<nfs-path>,"volume-opt=o=addr=<nfs-address>,vers=4,soft,timeo=180,bg,tcp,rw"'
    --name myservice \
    <IMAGE>

$ sudo docker run -d   --name devtest   --mount source=myvol2,target=/app   nginx:latest
8639e7cc80f422fdbc00b7209a3f976368af7692d38f75b4310b81961c27fc11

*inspect volume: *

$ sudo docker inspect myvol2
[
    {
        "CreatedAt": "2022-08-17T10:37:28+05:30",
        "Driver": "local",
        "Labels": null,
        "Mountpoint": "/var/lib/docker/volumes/myvol2/_data",
        "Name": "myvol2",
        "Options": null,
        "Scope": "local"
    }
]

How to Handle Storage in Docker?

bhargavirengarajan21 — Wed, 17 Aug 2022 03:07:00 +0000

Why should i need to store data?

Lets have a scenario here, we have our Mysql containers running and we try to fetch data in a application. While fetching the data, container stopped abbruptly. if you start container again and request for data. NOW ALL YOUR DATA IS GONE !

The life of the data depends on the container, once container is gone, data too is gone, even if another process needs it , bring back could be herculean task

2.Container Writeable layer is tightly coupled with host ,
where you can't move the data.

3.If need to store data , we can use storage driver to manage
file system, then again this extra abstraction will reduce
the performence

Which data should I back-up?

We need to back up data to a permanent storage. We have 2 layers of data. Read only layer(permanently stores data) and Read/Write data(Volatile). Obivous we need to back up R/W data.

Where Should to Backup ?

Docker provides storage objects:

Volume
Bind mount
temp fs

Volume:

Managed by Docker,Dedicated Directory in host's file system which are mounted on containers.
We can use Volume for multiple containers simulatneously, No automatic deletion of volumes, we need to delete if its not required.
It may be named or anonymous. Anonymous volumes are not given an explicit name when they are first mounted, so Docker provides unique random name withing the Docker host.Named volumes can persist data after we restart or remove a container. Also, it's accessible by other containers.
Volumes supports volume drivers, which allow you to store your data on remote hosts or cloud providers.

Volume:

Container Provides data and User provides commands to store/manage data to docker engine. But what container knows about is just name of the volume but not the path of the volume in the host. Even the external application having access to the container, wont be able to access Data stored in volume. Providing isolation and security for both host and containers

where can we use ?
1. Sharing data among multiple running containers.
2. When the Docker host is not guaranteed to have a
given directory or file structure.
3. When you want to store your container’s data on a
remote host or a cloud provider, rather than locally.
4. We can use for backup,restore or migration from one
host to another
5. Application requires high-performance I/O on Docker
Desktop. and also fully native files system.

Bind mount:
Its is very similar to Volume mounts, but with limited benefits. A file or Directory in host system is mounted on container. It is referenced by absolute path on host machine.

It is created on demand if not existed. Bind mounts useful in this case but it expects the host system to contain the sprcific directory structure. hence developer himself sometimes might not have that structure in his host.

since it exposes the storage location of the container, which can make dents on the overall security of application or host.

consider using named volumes. we can't use CLI commands to directly manage bind mounts.

where can we use ?

Sharing configuration file from host to containers. By this docker provides DNS resolution to containers, by mounting /etc/resolve.conf into each container.
When we share source code/build artifacts between a development environment host and a container. we may mount a app project/directory on a host, when u create same project everytime , it uses the built artifacts.

If you use Docker for development this way, your production Dockerfile would copy the production-ready artifacts directly into the image, rather than relying on a bind mount.

When the file or directory structure of the Docker host is guaranteed to be consistent.

tempfs
tmpfs a temporary file system.

Volumes and bind-mount allows you to share files between host and container, and data is persisted even container is stopped.On the other hand,tmpfs mount, only persists in the host's memory, not in storage. When the container stops, the tmpfs mount removed.

Only in Linux we have tmpfs mounts. When you create a container with tmpfs mount, the container can create files outside the container's writable layer. They can be created not shipped

example we can use this type of storage is like user session, browser history in incognito.

where to use ?

When we don't want to persist data, if we want data until the container is running.

Named pipes:
An npipe mount used for communication between the host and container.

where to use ?
This is used to run a third-party tool inside a container and connect to the Docker Engine API using a named pipe.

Docker network Commands (Docker Series - IV)

bhargavirengarajan21 — Tue, 16 Aug 2022 13:53:00 +0000

Some network commands:

$ sudo docker network create --driver bridge my-bridge
3973cea88d3f345c60ade7414dee4ff0a09f2863d667e713acef61dcf4badfa6

The above code is to create a network of type bridge with default Ip address.

$ sudo docker network create --driver bridge --subnet=192.168.0.0/16 --ip-range=192.168.5.0/24 my-bridge-1

eb05210ca8a0c722684271373dbdb7ea2da57dd82754324c59ffac363be22cd0

The above code is to create bridge network with specified ip range and subnet

$ sudo docker network ls

NETWORK ID     NAME          DRIVER    SCOPE
a2884fc4f337   bridge        bridge    local
6bffa8afea2f   host          host      local
3973cea88d3f   my-bridge     bridge    local
eb05210ca8a0   my-bridge-1   bridge    local
4278b3734e55   none          null      local

apart from created network , docker provides us 3 network, host, bridge, and none is a special case where its completely isolated and lack of connnectivity.

$ sudo docker network ls --filter driver=bridge
NETWORK ID     NAME          DRIVER    SCOPE
a2884fc4f337   bridge        bridge    local
3973cea88d3f   my-bridge     bridge    local
eb05210ca8a0   my-bridge-1   bridge    local

the above code helps to filter the network which has driver as bridge.

Connect Network:

$ sudo docker network connect my-bridge-1 flamboyant_wing

In the above code the we have established a bridge connection to running container. This wont return any id, to check we need to do
{% embed $ sudo docker inspect %} flamboyant_wing(container name)

result

Connect using network flag in run command:

$ sudo docker container run -itd --network host --name cont_ngnix nginx:latest
9227518678e384232b6e34f31674e5618f17851d0f49a317c33294faecec06e5

Here we provided the host network. when we try to find the port
sudo docker container port cont_ngnix
it returns nothing, since there is no port mapping as it's uses host ip address. instead try hitting the localhost 80 port which is host port.

Result on $ sudo docker inspect cont_ngnix

Inspect Network

 $ sudo docker network inspect my-bridge-1

[
    {
        "Name": "my-bridge-1",
        "Id": "eb05210ca8a0c722684271373dbdb7ea2da57dd82754324c59ffac363be22cd0",
        "Created": "2022-08-15T21:57:03.886178549+05:30",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": false,
        **"IPAM": {
            "Driver": "default",
            "Options": {},
            "Config": [
                {
                    "Subnet": "192.168.0.0/16",
                    "IPRange": "192.168.5.0/24",
                    "Gateway": "192.168.5.0"
                }
            ]
        },**
        "Internal": false,
        "Attachable": false,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
       ** "Containers": {},**
        "Options": {},
        "Labels": {}
    }
]

the above command inspect user created network displays the related details.

format inspected details:

$sudo docker network inspect --format "{{.Scope}}" bridge

the above command helps to list the scope of bridge network

$sudo docker network inspect --format "{{.Id}}:{{.Name}}" bridge

f46032790f9dcb26f99afec265b1eea44fc96563836a85316c4346f36aa2c6fe:bridge

this command list the id and name of the bridge network

Disconnect Network

$sudo docker network disconnect my-bridge-1 flamboyant_wing

How Docker Communicates ? (Docker Series - IV)

bhargavirengarajan21 — Mon, 15 Aug 2022 16:05:00 +0000

This is a learner post so if anything needs to be corrected or updated, kindly comment. will be helpful to know !.

When our application contains more than one container. For instance a food application might need a frontend, backend, databases based container.
We need to exchange information between containers. we need a network to carry out crucial data between containers. communication can be one to one or many to one.

How do they manage ?

Network Drivers!!

Piece of software , that handles container networking. created by network command.
invoking network inside the cluster or host
docker itself uses these network drivers to communicate with other container. So it provide native network driver. We can even use 3rd party drivers for specific use case.

Container Networking model:

Host N/W infrastructure: includes software and hardware infrastructure Wifi , ethernet and host kernel network stack.
IPAM manages IP address within context
Docker Engine creates Individual network object. user created and default container network.

If a container contains more than one network , it will have more than one endpoints and different set of IP address.
When its single host implementation, scopes are limited.
Within the same scope, if two containers are connected to same network it can communicate via DNS where container names can be used instead of IP. These informations are provided by IPAM. Then the driver and IPAM convert the request as host supported network packets. We need to make sure the containers communicate outside, if not simple apt-get command won't execute.

Types of Native Network ?

Host Network:
The container is not isolated from the docker host. If you run container that binds with ports 80 and if we use host networking then the container is available in port 80 of host IP address.

This is used for handling large number of ports, where we require NTA or proxy.
But this type of network is available only for Linux, not for windows or Mac.

Bridge Network:

Default for docker container. If we don't specify any network it will assign to default bridge network.
This bridge network connect containers with virtual ethernet and communicates to host via this ethernet. Container will have different IPs as the host is isolated from the container. The container and host communication within only same bridge scope.
We can provide IP range and subnet mask for the bridge network, or the IPAM will manage the ip address for us, we can ping the address provided in the virtual bridge.

Overlay Network:
This does not apply for single host infrastructure. This applies where there is more than one docker host daemon or containers. Hence keeping track of container Ip is not enough , as we need to communicate with container and host, so we will maintain 2 layer of information.
overlay information - Ingress:
connect the data and control of the source and destination Container IP. Docker Swarm data and network traffic is handled by ingress network. Hence use this information by default
underlay information- docker_gwbridge:
Data regarding the source and destination host IP. Docker Swarn , it is bridge network which connects the individual Docker daemon to the other daemons.

MacVlan Network:(virtual local area network) It is a virtualisation , where we have multiple mac of a physical interface which has its own ip address. This type can be used for application whch monitors the network traffice where it is directly connected to physical interface.
Advantage of using macvlan is performence is better than overlay and does not linux bridge*.
things to be noted before using mcvlan

unintentional damange to the physical network since it is directly connect.
able to handle "promiscuous mode" where one physical network is assigned to multiple mac address.
if your application works in both overlay and bridge, then they should be long term.

IpVlan Network: same as macvlan but endpoint has same mac address,has 2 mode l2 and l3. L2 where end points has same mac address different ip address , L3 packets are rounted between endpoints.
uses of Ipvlan
ipvlan should be used in cases where some switches restrict the maximum number of mac address per physical port due to port security configuration.

Linux bridge* - it is like a regular hardware switch with learning and also supports protocols like STP for loop prevention. Here VMs or Containers will connect to bridge and bridge will connect to outside world. For external connectivity, we would need to use NAT.

What is Dockerfile ? (Docker Series - III)

bhargavirengarajan21 — Mon, 15 Aug 2022 09:17:00 +0000

If anything missed kindly add in comment

A sequential set of instruction for Docker engine , or a set of commands to assemble the an image
Primary way of interacting with Docker and migrating containers.
Each sequential instructions are processed individually and results are stored as stack of layers. This sequential layer managed by file system becomes a docker image.
Sequential layers helps to cache and helps to troubleshoot.This precreated layer are reused by docker daemon when we run two docker file that has same layer at some stage.

Format:

ARG
FROM
RUN
ADD|COPY
ENV
CMD
ENTRYPOINT
EXPOSE

No extention required, can be created from any text editor, its good to start a docker file name with 'D' capital.

EXAMPLE:

ARG VERSION=16.04
FROM Ubuntu:${VERSION}
RUN apt-get update -y
CMD ["bash"]

Comments in Dockerfile is represented using '#'. But all hash preceeded sentence is not comments. That also means parser directive.

Parser directive
parser directive, one of the docker command indicates, how the docker file should be handled or read. It should be at the top of the file.

Types of parser:

syntax: Its only available in buildKit, but its used
to know the location of the dockerfile.
escape: it available anywher, it used to specify the
character to escape. default is '\'.

COMMANDS:

ARG is the only instruction that may precede FROM
FROM can appear multiple times within a single Dockerfile to create multiple images or use one build stage as a dependency for another.
WORKDIR This sets the working directory for Run, cmd, copy. By default the WORKDIR is created if not specified.
RUN to execute any command in new layer on top of current layer RUN ["executable", "param1", "param2"]. (exec form) another way of executing command irrespective of the shell based. eg RUN ["/bin/bash", "-c", "echo hello"].
ADD Add new files from src and copies to dest. used to copy to docker image. should be used with caution.
COPY Copy new files from src to dest, used to copy to docker container.
Env Provide value for environment variable
CMD Used to execute command in shell docker file can contain only on CMD , if more than one specified last final will be executed.
Entry point Used to execute command when container is initiated
Expose expose a particular port with specific protocols

How Docker Works ? (Docker Series - II)

bhargavirengarajan21 — Sat, 13 Aug 2022 17:04:00 +0000

Docker is a Client- Server architecture.

What does the docker client do ?

The client is a tool for users to communicate/ interact with Docker. We can use commands and also Docker provided API to communicate.

What is docker host and what does docker host contain ?

The docker host environment to run or execute your application. It runs a program or a piece of software called Daemon. It communicates bi-directionally where it gets user-input and prints the response received.

contains ?:

Docker Daemon: This manages docker objects such as images and containers. Daemon builds the docker file and processes it as an image.
Docker Image: Read-only template, contains instructions to build a container. This can be either own image or pulled existing image from docker registry. Docker images are built from a docker file. We can publish our own image in docker. Docker image communicates directly to Daemon. When redeploying an image it rebuilds where the image has changed, so it's light-weight and faster than vm technology.
Docker Container: Runnable instance of docker image, we can create, stop, run using cli or api. Containers are attached to networks and storage. By default it runs isolated from its host machine. The container communicates to daemon through images.

Docker Registry ?

Docker registry is a repository to store the images. We can also maintain a private registry. Docker will look for the images from the registry by default.

We can pull or run images that are available in the registry as well as push images.

What is Docker ? (Docker Series - I)

bhargavirengarajan21 — Sun, 10 Jul 2022 10:35:00 +0000

Docker is a open platform to package your code ,run, deploy, ship anywhere.

Why docker ?
Let's say my application stack contains of MongoDB, Redis, NodeJ. Are we sure that these libraries and dependencies compatabile with the OS we are using ?("Matrix from hell"). Herecomes the docker, it seperates the application running from the infrastructure.

How docker does that ?
Containers !!. Docker does that with the help containers.
It is a loosely isolated environment , light-weight, runnable instance. Isolation and security lets us run many containers on the same host by sharing the os kernal.

The above issue can also be handled by virtualisation, We need to create seperate VM instance for each dependancies to run which could be a overhead to maintain.

Docker Container vs Virtualisation

What I can use Docker for?

Fast and consisten delivery of application:
Responsive Deployment, portable and scaling
Running more workload on same hardware