DEV Community: Ramya Perumal

RAG - Memory Systems

Ramya Perumal — Thu, 13 Aug 2026 18:08:59 +0000

We need memory to store the previous conversational history.

For example:

Previous question is:

User: File handling in Python

Assistant: Explain about file handling.

Next time, the user asks:

User: What are the modes in it?

The LLM should understand the context and then respond.

Below are the details that can be stored in the memory.

Past Conversations
User Preferences that we specify in the system prompts, e.g., JSON format
Past Decisions
Previous Tasks

Based on the details we are storing, we will choose between long-term or short-term memory.

Short-Term Memory

Redis, Valkey, Memcached, and Cosmos, which are cached databases, can be used for short-term memory to store the last few conversations or a summary.

We can set a general data invalidation rule to erase the content or use an LRU cache eviction policy, where the least recently used data will be erased from the database.

Long-Term Memory

Postgres, Pinecone, and MongoDB can be used for long-term memory to store long conversation histories.

How It Is Functioning

A summary of the entire conversation history will be stored in short-term memory to reduce latency whenever needed.

Episodic Memory

Episodic memory is a type of memory that stores specific events or experiences that happened in the past, usually together with information about what happened, when it happened, and the context surrounding it.

We can use either a short-term or long-term memory database depending on the use case. It is a kind of combination of short-term and long-term memory.

User:

I am planning a trip to Paris.

Agent:

How many days will you stay?

User:

5 days.

Later,

User:

Can you suggest an itinerary?

Episode 1

User wants to travel to Paris.

Trip duration: 5 days.

User previously mentioned:

Destination = Paris

Duration = 5 days

This information can be used to provide a more relevant response.

This helps the LLM understand what happened previously.

Semantic Memory

Semantic memory contains facts extracted from previous conversational history. Semantic memory is generally considered long-term memory.

Example

Paris is the capital of France.

The Louvre is a museum in Paris.

France uses the Euro.

That's general knowledge.

Sliding Window Memory

It is a short-term memory. Here, we store the last 3 to 4 conversations.

Redis or Valkey, like any cache memory, can be used.

Summarized Memory

Each and every time, the conversation, which includes the user query and response, will be summarized.

Even though token consumption during summarization is more, overall token consumption will be less.

It is a long-term memory.

Entity Fact Memory

This memory is used to store facts about a particular entity.

The difference between entity fact memory and semantic memory is that semantic memory is the broader category. Entity fact memory is one way of organizing and storing semantic knowledge about specific entities.

It can be used as either long-term or short-term memory.

Semantic Memory

Entity: Python

Python → is a programming language

Python → is used for AI

Python → supports object-oriented programming

Python → was created by Guido van Rossum

Entity Fact Memory

Entity: Alice

Alice → works at ABC Company

Alice → prefers Python

Alice → is working on Project X

It is not a good practice to store the entire conversation. We can make decisions based on the conversation and then store the relevant information. This is a good practice.

RAG - Multivector Retrievel, Multi Hop, Conversational RAG

Ramya Perumal — Wed, 12 Aug 2026 17:51:51 +0000

Multi-Vector Retrieval

In a typical RAG pipeline, every chunk is converted into an embedding and stored in the vector database.

In Multi-Vector Retrieval, instead of creating a single embedding for a chunk, multiple embeddings are created to represent different aspects of the original chunk.

Each embedding captures a different perspective of the same content. The metadata of each embedding contains a reference to the original chunk.

When a user query is received, the retriever searches across all these embeddings. Since each embedding represents a different aspect of the content, the chances of retrieving more relevant information are higher.

This approach generally provides better retrieval performance.

However, it is a costly approach because:

Multiple embeddings are generated for every chunk.
More storage is required in the vector database.
More tokens are consumed during embedding generation.

If no other optimization technique is able to improve the RAG performance, Multi-Vector Retrieval can be considered as a final optimization step.

Multi-Hop

Multi-Hop Retrieval is used when the answer cannot be obtained from a single piece of context. Instead, the LLM has to retrieve multiple related contexts and connect them to generate the final answer.

Example

Suppose the knowledge base contains the following information:

Biryani contains spices.
The spices used in biryani are cardamom, cinnamon, cloves, etc.

Now suppose the user asks:

"What spices need to be added to biryani?"

The LLM first retrieves the information:

"Biryani contains spices."

It then hops to the next related context:

"The spices used in biryani are cardamom, cinnamon, cloves, etc."

Finally, it combines both pieces of information to generate the complete answer.

In this process, the LLM hops from one retrieved context to another until it gathers enough information to answer the user's query.

Each retrieved context should have a meaningful relationship with the next one so that the LLM can follow the chain of information and produce the correct response.

Conversational RAG

A user may not restrict themselves to asking only one query. They may ask a sequence of queries. Here, we are going to look at how to build Conversational RAG.

Traditional Flow

User Query
Retriever
Related Documents
LLM
Answer

A user may ask questions that are indirect or related to the previous question. So, we need to implement Query Transformation and Expansion to get the related queries and their responses.

Therefore, we need to have memory to store the previous conversation history.

Conversational RAG will be:

Conversation History + Current Question
Intent
Query Transformation and Expansion
Retriever
Relevant Context
LLM
Context-Aware Answer

Here, we are going to see how we can store conversation history in memory.

We can use persistent memory such as PostgreSQL or SQLite, or short-term memory using Redis, Valkey, Memcached, etc. We can also use a pickle file to store the history.

If we are using a file, make sure that the application we are using is single-threaded.

To store the history, we can use the session ID as the primary key.

Always store the entire history in long-term memory. However, we cannot store the entire history in short-term memory.

The purpose of storing history in short-term memory is to reduce latency.

To solve this problem, we can store the entire history in long-term memory and summarize the conversation history and store the summary in short-term memory whenever needed.

RAG - Parent Retriever

Ramya Perumal — Wed, 05 Aug 2026 01:52:04 +0000

RAG Pipeline Optimization Techniques

The following techniques are commonly used to optimize a RAG pipeline:

Semantic Caching
Query Transformation and Expansion
Context Compression
Parent Retrieval
Multi-Vector Retrieval

These techniques are used to improve the quality of the responses generated by the LLM.

Parent Retrieval

Today, we are going to focus on Parent Retrieval.

A chunk is a small portion of a document or paragraph, typically consisting of 500–1000 words. When these chunks are converted into embeddings and stored in a vector database, semantically related chunks are positioned close to one another.

When a user submits a query, the system retrieves the chunks that are closest to the query. However, not all the retrieved chunks may be the most relevant. Sometimes, we may miss other chunks that provide better context for the user's query.

Example

Suppose we have the following paragraphs:

Paragraph 1 (P1)

Each line is stored as an individual chunk:

P1C1
P1C2
P1C3
P1C4

Paragraph 2 (P2)

Each line is also stored as individual chunks:

P2C1
P2C2
P2C3

Suppose the expected answer to the user query is P1C2.

The vector database retrieves:

P1C2
P2C3

Although P1C2 is correctly retrieved, the related chunks P1C1, P1C3, and P1C4 may provide much better context than P2C3. Since these chunks are not retrieved, we may lose important context that could improve the final LLM response.

If we store the document paragraph-wise instead of using smaller chunks, we may retrieve unnecessary context, which increases token consumption.

Solution – Parent Retrieval

Suppose Paragraph 1 is divided into four chunks:

P1C1
P1C2
P1C3
P1C4

Whenever a specific chunk is retrieved from the vector database, it also contains a reference to its parent paragraph, which is stored in the chunk's metadata.

For example, if P1C2 is retrieved, the metadata also contains a reference to Paragraph 1.

Using this reference, the retriever can also fetch the remaining chunks belonging to the same parent paragraph:

P1C1
P1C3
P1C4

This ensures that we do not miss important context that is closely related to the user's query.

Example

Suppose the user query is:

"Create a dictionary"

The retrieved chunks are:

P1C2
P2C2
P2C3

Instead of sending only these chunks to the LLM, Parent Retrieval also includes the remaining chunks from the same parent paragraph:

P1C1
P1C3
P1C4

Providing this additional context enables the LLM to generate a more accurate response.

When Should Parent Retrieval Be Used?

Parent Retrieval is useful when we feel that important context is missing from the retrieved chunks and we want to include additional context that is closely related to the query.

The parent paragraph is stored similarly to other chunks. However, it is not used directly for vector search. Instead, its reference is stored in each chunk's metadata, allowing the retriever to fetch the parent context whenever required.

RAG - Context Compression

Ramya Perumal — Thu, 30 Jul 2026 01:58:52 +0000

In a RAG pipeline, documents are first split into chunks, converted into embeddings, and stored in a vector database.

When a user query arrives, it is converted into an embedding and used to retrieve relevant results from the vector database. Although the retrieved results may be relevant, they may not always be directly related to the user's query.

Example

User Query

"FastAPI dependency injection"

The retrieved results may include:

"Use FastAPI in deployment"
"How to dockerize a FastAPI application"

These results may be semantically related to FastAPI, but they are not necessarily the most accurate results for the user's query. This depends on how the documents were embedded and stored in the vector database.

To improve the quality of the retrieved context, the fetched results can be compressed into one or two chunks before they are passed to the augmentation phase. This reduces the number of tokens used during augmentation and acts as an optimization technique.

From the above example, the chunks retrieved from the vector database may not be completely relevant to the user query. Therefore, we compress the retrieved chunks without changing their original context and create one or two chunks that are more relevant to the query.

Context compression is not a mandatory step in every RAG pipeline. It is generally adopted based on trial and error and depends on the application's requirements.

This compression technique is mainly useful when:

There are token constraints.
Higher-quality retrieval results are expected.

Context Compression Techniques

There are different approaches for context compression:

LLM-based Compression
Embedding-based Compression
Keyword-based Compression

1. LLM-based Compression

In this approach, the relevant chunks retrieved from the vector database are given to an LLM for compression.

For example, suppose four chunks are retrieved from the vector database. Instead of sending all four chunks to the main LLM, they are first passed to another LLM whose responsibility is only to compress them into one or two meaningful chunks.

Although using an LLM for compression may appear expensive, a locally deployed LLM or a low-cost LLM is typically used for this task.

The purpose of this LLM is not to generate the final response. Its responsibility is only to merge and compress the retrieved chunks while preserving their context.

The compressed chunks are then sent to a more powerful or application-specific LLM to generate the final response.

This creates a two-stage pipeline, which is more token-efficient while still producing high-quality output.

2. Embedding-based Compression

In embedding-based compression, the embeddings of the top retrieved chunks are compared with the embedding of the user query using cosine similarity.

The chunks that are most relevant to the user query are selected and used as the compressed context.

3. Keyword-based Compression

Keyword-based compression uses techniques such as TF-IDF or the BM25 algorithm.

The keywords present in the user query are compared with the retrieved chunks.

The chunks that have the highest keyword relevance are selected and used as the compressed context.

Combining Compression Techniques

A single application is not limited to using only one compression technique.

Depending on the application requirements, two or more context compression techniques can be combined to improve retrieval quality while reducing token usage.

RAG - Query Transformation and Expansion

Ramya Perumal — Wed, 29 Jul 2026 17:27:33 +0000

Query Transformation

When a user asks a blended or incomplete query, such as:

"How do I deploy it?"

the LLM uses the background context it already has to transform the query into a more meaningful one.

For example:

"How do I deploy a FastAPI application?"

By transforming the query into a more specific one, the system is able to retrieve more relevant documents from the vector database.

For query transformation to work effectively, the LLM should have some background information about the conversation.

Query Expansion

When a user query is converted into an embedding, the point obtained in the vector database may not be close to the most relevant documents.

By expanding the query into different variations, we can retrieve more relevant documents.

Example

Original User Query

"How do I deploy a FastAPI application?"

Expanded Queries

Deploy FastAPI using Docker
FastAPI deployment guide
FastAPI deployment on AWS
FastAPI with Gunicorn
FastAPI with Uvicorn

These are different variations of the original user query.

How Does Query Expansion Work?

For each variation of the user query, the system retrieves a set of relevant contexts from the vector database.

During the augmentation phase, the retrieved contexts, along with the original user query, are sent to the LLM so that it can generate more accurate results.

Query Expansion and Query Transformation

Both query expansion and query transformation are performed by the LLM.

Query Transformation rewrites an incomplete or ambiguous query into a more meaningful query.
Query Expansion generates multiple variations of the user query to improve document retrieval.

Alternative Approach

Instead of calling the LLM for query expansion, we can use a rule-based approach by storing related queries for a user query as a cluster in the vector database.

When a user submits a query, the system retrieves the related queries from the cluster and uses them to improve document retrieval.

RAG - Semantic Caching

Ramya Perumal — Sat, 18 Jul 2026 17:12:20 +0000

When a user submits a query, the query is converted into an embedding and searched against the vector database to retrieve the relevant documents.

But what happens if the user asks the same or a very similar query again?

This is where semantic caching comes into the picture.

Instead of searching the vector database again, the system stores the previous search result in a cache. A cache is a temporary storage where frequently accessed or recently queried results are stored. When the user asks the same or a semantically similar query again, the system can retrieve the result directly from the cache instead of querying the vector database again.

Benefits

Saves retrieval time
Reduces token consumption
Reduces the number of calls to the vector database
Reduces the number of calls to the LLM

How Do We Store Results in the Cache?

We can use Redis or Valkey for semantic caching.

These are in-memory databases, which means they store data in RAM instead of disk. Since data is stored in memory, retrieval is much faster compared to traditional databases.

Typically, we store:

User query
Related answer
Metadata
Embeddings

Example

Suppose a user asks:

"What is today's gold price?"

The query and its corresponding answer are stored in Redis.

Later, another user asks:

"Gold price today?"

Although both queries have the same meaning, Redis cannot directly retrieve the previous answer because it expects the key to match exactly.

This is one of the limitations of using Redis as a simple key-value store.

How Can We Solve This?

One approach is:

Retrieve all the keys stored in Redis (for example, using KEYS *).
Generate or retrieve the embedding for each stored query.
Convert the current user query into an embedding.
Compare the current query embedding with the stored query embeddings using cosine similarity.
If the similarity score is above a predefined threshold, retrieve the corresponding answer from Redis.

This allows semantically similar queries to reuse cached results even when the text is different.

Ways to Implement Semantic Caching

Semantic caching can be implemented in two ways:

Using frameworks such as LangChain
Using in-memory databases such as Redis, Valkey, or other similar databases

Cache Invalidation

One of the most important aspects of semantic caching is cache invalidation, which determines how long cached data should remain valid before it is automatically removed or refreshed.

For example, suppose a user asks:

"What is today's gold price?"

The answer should only be valid for a limited period. If the application returns yesterday's gold price, the information becomes incorrect.

There is no single solution for cache invalidation. The appropriate strategy depends on the application and the type of data being cached.

Different scenarios need to be considered before deciding when cached data should expire.

When Should In-Memory Databases Be Used?

In-memory databases are well suited for:

Temporary queries
Frequently asked questions
Data that is accessed repeatedly

By understanding the meaning of the query, we can define guardrails to determine which queries should be cached and when the cache should be invalidated.

The main objective is to optimize the RAG pipeline by reducing unnecessary calls to both the vector database and the LLM.

Although it is not possible to eliminate duplicate requests completely, semantic caching can significantly reduce them.

Important Consideration

We should not store every query in an in-memory database.

Only queries that are valuable for caching should be stored because RAM has limited storage capacity. Therefore, an effective caching strategy should carefully decide which queries are worth storing and for how long.

RAG - Meta Filtering and Reranking

Ramya Perumal — Sun, 12 Jul 2026 21:43:55 +0000

Generally, when a user asks a query, the system searches for the relevant chunks stored in the vector database using cosine similarity. The better we can filter the data, the smaller the search space becomes, resulting in faster and more efficient retrieval.

Suppose we have a book with 10 chapters. If we want to search for a particular topic, all the points in the vector database are compared with the user query, and only the closest points are retrieved. This process is called KNN (K-Nearest Neighbors).

Another algorithm is ANN (Approximate Nearest Neighbors). Instead of checking all the points in the vector database, ANN searches only within a smaller region based on the proximity of the data. As the name suggests, it does not always return the exact result, but it provides the most preferred or approximate results much faster.

Is there any other method we can use to make the search more effective?

Metadata Filtering

Metadata means data about the data.

Metadata is stored along with each chunk. It can contain information related to the chunk, such as the chapter name, topic description, author, or any other relevant details.

When the user query contains information related to the metadata (for example, a chapter name or topic), the system can directly filter the relevant chunks before performing vector similarity search. This technique is called metadata filtering.

Metadata filtering is supported by:

Pinecone
ChromaDB
Qdrant

FAISS does not provide built-in support for metadata filtering.

Reranking

Documents are first split into chunks, and each chunk is converted into vectors and stored in the vector database.

When a user query arrives, it is converted into a vector and searched against the vector database to retrieve the closest chunks. However, we do not know whether the retrieved documents are actually the most relevant to the query. It is not always true that the closest vectors represent the most relevant documents.

How Reranking Works

The documents retrieved from the vector database are passed to a cross-encoder along with the user query.

The cross-encoder assigns a relevance score that indicates how closely each document matches the query. The documents are then displayed in ascending or descending order based on these scores.

The results produced by the cross-encoder are called reranked results.

The retrieved documents remain the same as those returned by the vector database, but their order changes. Documents with higher relevance scores appear before those with lower scores.

A cross-encoder is a neural ranking model. Instead of encoding the query and documents separately, it takes both the query and the document together as input to a transformer model and generates a relevance score for each document.

There are transformer models specifically designed for reranking tasks. The encoder understands the meaning of both the query and the document and reranks the documents accordingly.

Why Use Reranking?

Reranking is an important step in the RAG pipeline.

It is especially useful when working with documents that contain images or other multimodal content.

Example

Suppose the user asks:

"Show me the front view of the truck."

The vector database may retrieve multiple images related to trucks because they are semantically similar.

The reranker analyzes both the query and the retrieved images (or their associated text descriptions) and assigns relevance scores.

As a result, the image showing the front view of the truck receives a higher score than the other truck images, making it appear first in the final results.

Docker -Networking and Best Practices

Ramya Perumal — Sat, 27 Jun 2026 20:30:19 +0000

Docker Networking

Containers are assigned an IP address when they are created. To check the IP address, we can use the following command:

docker inspect <container_id>

If we send a request from the host to the container's IP address, the container responds using its assigned IP address.

By default, Docker creates a bridge network. This bridge network allows:

Communication between the host and the container (through port mapping).
Communication between containers connected to the same bridge network.

With the default bridge network, containers generally communicate using IP addresses. To communicate using container names (hostnames), we can use a custom bridge network.

The main difference between the default bridge network and a custom bridge network is that containers on a custom bridge network can communicate using their container names (DNS resolution), making it suitable for production environments.

When to Use a Custom Bridge Network

A custom bridge network is useful when an application consists of multiple services running in separate containers. These containers can communicate with one another using their container names instead of IP addresses.

Create a Custom Bridge Network

docker network create mybridge

Create and Run Containers Inside the Network

docker run -it --network mybridge --name container1 busybox:1.36 sh

docker run -it --network mybridge --name container2 busybox:1.36 sh

Now, from container1, run:

ping container2

Similarly, from container2, run:

ping container1

If the ping is successful, it confirms that both containers can communicate because they are connected to the same custom bridge network.

Example Workflow for a Docker Bridge Network

Step 1: Build the Images

docker build -f Dockerfile -t flask_app:v1 .
docker build -f httpd.Dockerfile -t apache_container:v1 .

Step 2: Create the Network

docker network create bridge_app

Step 3: Run the Containers

docker run -d --name flask_new --network bridge_app flask_app:v1

docker run -d --name apache_new --network bridge_app apache_container:v1

Step 4: Verify the Network

docker network inspect bridge_app

Step 5: Test Communication

ping apache_new
ping flask_new

You can also use curl with the application's port to access another container's application.

Host Network

Suppose we run an application inside a Docker container.

If we do not expose the application port using -p, the application cannot be accessed through the host machine. It can only be accessed using the container's IP address (if reachable).

Example:

docker run flask_app:v1

If we expose the port:

docker run -p 5000:5000 flask_app:v1

Docker maps the host port to the container port, allowing the application to be accessed using the host machine's IP address.

Host Network Mode

docker run --network host flask_app:v1

In host network mode, the container shares the host's network stack. Therefore, the application can use the host's network directly without explicit port mapping.

Note: Host networking is supported on Linux. On Docker Desktop for Windows and macOS, host networking is limited and generally not recommended. Port mapping (-p) is the standard approach.

Docker Image Optimization

Why Is Image Optimization Needed?

Smaller images start containers faster.
Smaller images are easier to share.
Smaller images consume less storage.
Smaller images download faster.

1. Multi-Stage Builds

In a multi-stage build, the first stage builds the application, and the second stage copies only the required artifacts into the final image.

A single-stage build includes both build tools and runtime dependencies, making the image larger.

A multi-stage build keeps only the files required to run the application.

Single-Stage Build

FROM python:3.9-slim

COPY . /app

WORKDIR /app

CMD ["python", "main.py"]

Multi-Stage Build

FROM python:3.9-slim AS builder

WORKDIR /app

COPY main.py .

FROM python:3.9-slim AS runner

WORKDIR /app

COPY --from=builder /app/main.py .

CMD ["python", "main.py"]

Here:

Builder stage prepares the application.
Runner stage copies only the required files, resulting in a smaller final image.

2. Choose a Minimal Base Image

Using a lightweight base image reduces the final image size.

Examples:

FROM python:3.9-slim

This image contains only the essential Python packages.

FROM python:3.9-alpine

This image is even smaller and includes only minimal functionality. Additional packages must be installed separately.

It is not recommended to use:

FROM python

because Docker will pull the latest version, which may introduce compatibility issues. Always specify a version tag.

3. Layer Caching

Docker caches image layers.

If no changes occur in a layer or any previous layer, Docker reuses the cached layer, making builds much faster.

The order of Dockerfile instructions is important.

If COPY . /app is placed near the beginning of the Dockerfile, any source code change invalidates all subsequent layers.

Instead, place frequently changing instructions lower in the Dockerfile whenever possible.

Less Efficient

FROM python:3.9-slim

COPY . /app

WORKDIR /app

CMD ["python", "main.py"]

Better

FROM python:3.9-slim

WORKDIR /app

COPY . /app

CMD ["python", "main.py"]

This prevents Docker from rebuilding the WORKDIR layer unnecessarily.

4. Run Containers as a Non-Root User

By default, containers run as the root user.

A root user can create or modify files anywhere inside the container.

If the container is compromised, an attacker may gain elevated privileges.

Running the container as a non-root user improves security because that user has limited permissions.

Example (Windows Containers):

RUN icacls C:\app /grant ContainerUser:(OI)(CI)F

USER ContainerUser

Example (Linux Containers):

RUN useradd -m appuser

USER appuser

Running applications as a non-root user is considered a Docker best practice.

Interview Questions

Question 1

Which Docker network allows containers to communicate with each other without requiring port mapping on the host?

Answer: Bridge network.

Question 2

Which statement is true about Docker layer caching?

Answer:

Docker caches layers unless a previous layer has changed.

Question 3

Why is it considered best practice to run containers as a non-root user?

Answer:

It helps prevent attackers from gaining root privileges if the container is compromised.

Question 4

What is the purpose of using a multi-stage build in Docker?

Answer:

To reduce the size of the final Docker image.

Question 5

How do you specify a non-root user in Docker?

Answer:

Use the USER directive in the Dockerfile or the --user option when running the container.

Question 6

Which of the following can invalidate the Docker build cache for a layer?

Answer:

Changing the base image version.
Modifying files copied into that layer.
Adding or changing an environment variable (ENV) or build argument (ARG) used by that layer.
Changing the Dockerfile instruction itself.

Docker – ARG Directive, .dockerignore, and Docker Volumes

Ramya Perumal — Sat, 27 Jun 2026 20:30:07 +0000

ARG Directive

The ARG directive acts like a variable. We can define it inside the Dockerfile and change its value during the image build process.

ARG PYTHON_VERSION=3.8
FROM python:${PYTHON_VERSION}-slim

Here, the Python version in the Dockerfile is set to 3.8. However, during the build process, we can change it to 3.10.

docker build -f Dockerfile --build-arg PYTHON_VERSION=3.10 -t helloworld_flask:v1 .

-t means tag.
-f means Dockerfile path.

We can use ARG to make base image versions and other build-time values configurable.

Example

FROM node:20-alpine

# 1. Define the arguments
ARG APP_DIR=app
ARG INSTALL_ARGS="--omit=dev"

# 2. Use them in instructions
WORKDIR /${APP_DIR}
COPY . .
RUN npm install ${INSTALL_ARGS}

Note: ARG values can only be changed during the image build process. They cannot be changed during container creation.

Docker Ignore

A .dockerignore file is used to specify files and directories that should not be copied into the Docker build context.

Create a file named .dockerignore in the application's root directory.

Examples of files and folders that can be ignored:

Dockerfile
.venv
__pycache__
*.pyc
requirements.txt
.git
.gitignore

Ignoring unnecessary files reduces the build context size and speeds up image builds.

Docker Volumes

Generally, when a container is created, a writable layer is also created.

If we create files inside the container, they are stored in the writable layer. However, when the container is deleted, all data in the writable layer is lost.

What if we need to store files permanently on the host machine?

This is where Docker volumes come into the picture.

Docker volumes allow data to persist independently of the container lifecycle.

When a volume is mounted between a host directory and a container directory:

Files created in the container appear on the host machine.
Files created on the host machine appear inside the container.
Changes are synchronized between both locations.

Types of Docker Volumes

Bind-Mounted Volumes
Docker Managed Volumes (Named Volumes)

1. Bind-Mounted Volumes

A bind mount creates a mapping between a host directory and a container directory.

docker run -it -v ./data:/data busybox:1.36 sh

Here:

./data = Host machine directory
/data = Container directory

Characteristics:

Tightly coupled with the host file system.
Multiple containers can share the same host directory.
Changes made in either location are reflected in the other.

Note: The host directory is not deleted when the container is removed.

2. Docker Managed Volumes (Named Volumes)

Create a Docker-managed volume:

docker volume create dockersession

This creates a volume outside the container lifecycle.

List Volumes

docker volume ls

Inspect a Volume

docker volume inspect dockersession

This displays information about the volume, including its mount location.

Example Linux location:

/var/lib/docker/volumes/dockersession/_data

Mount the Volume to a Container

docker run -it -v dockersession:/data123 busybox:1.36 sh

Here:

dockersession = Volume name
/data123 = Container directory

Multiple containers can use the same volume for data sharing.

Find Containers Using a Specific Volume

docker ps -a --filter volume=dockersession

One of the major benefits of Docker volumes is that they are completely decoupled from the container lifecycle.

When a container is deleted, the volume and all its data remain safely stored on the host machine.

Interview Questions

Question:

What is the primary use of the ARG instruction in Docker?

Answer: To pass build-time variables to the Dockerfile.

Question:

Which of the following is true about ARG variables in Docker?

Answer: They are used only during the image build process.

Question:

Can an ARG variable be used in a RUN instruction within a Dockerfile?

Answer: Yes, but only after it has been declared.

Question:

Which files can be ignored using .dockerignore?

Answer: Any file or directory within the build context.

Question:

What is the purpose of Docker volumes?

Answer: To store data that persists even after a container is destroyed.

Question:

What is the default location of Docker volumes on Linux systems?

Answer:

/var/lib/docker/volumes

Question:

Which command allows you to list all Docker volumes?

Answer:

docker volume ls

Question:

In which scenario would you use a bind-mounted volume?

Answer: When you need to share specific directories between the host machine and a container.

Docker – Port Mapping, Logs, Container Management, and Image Removal

Ramya Perumal — Sat, 27 Jun 2026 20:29:51 +0000

Docker – Logs, Remove, and Port Mapping

Port Mapping

When we run an application inside a container, we define the port on which the application will run. The container runs the application on that port.

It is not possible to access an application running inside Docker from outside the container unless port mapping is configured.

Port mapping is specified using the -p option.

Syntax:

docker run -p <host_port>:<container_port>

Example:

docker run -p 8888:8080

Here:

8888 is the host (machine) port used to access the application from outside the container.
8080 is the port defined in the application and exposed inside the container.

If the host port is already in use, Docker displays an error message.

How to Check Which Ports Are Being Used by Docker Containers

docker ps --format "table {{.ID}}\t{{.Names}}\t{{.Ports}}"

Detached Mode

To run a container in detached mode, use the -d option.

docker run -d -p 2000:2001

Detached mode means the container runs in the background.

Docker Logs

Logs contain information about the activities happening inside a container.

View Logs of a Specific Container

docker logs <container_id>

Follow Logs Continuously

docker logs -f <container_id>

-f stands for follow.

View Logs from a Specific Time Range

Seconds (s): docker logs --since 30s <container_id>
Minutes (m): docker logs --since 5m <container_id>
Hours (h): docker logs --since 2h <container_id>

Docker supports seconds (s), minutes (m), and hours (h) for relative time.

For days, weeks, months, or years, use an ISO 8601 date or timestamp.

Weeks:  docker logs --since 2026-06-14 <container_id>
Months: docker logs --since 2026-05-21 <container_id>
Years:  docker logs --since 2025-06-21 <container_id>

Combined Time Units

docker logs --since 1h30m <container_id>

Displays logs from 1 hour and 30 minutes ago.

Exact Time

docker logs --since "2026-06-21T17:30:00" <container_id>

Displays logs generated since the specified date and time.

Docker Inspect

docker inspect is used to view detailed information about a container, image, network, or volume.

docker inspect <container_id>

Access a Running or Exited Container

To open a shell inside a running container:

docker exec -it <container_id> sh

-i = Interactive mode
-t = Allocate a terminal

Docker Remove

Delete an Exited Container

docker rm <container_id>

Delete a Running Container

docker rm -f <container_id>

-f forcefully stops and removes the container.

Alternative Method

First stop the container:

docker stop <container_id>

Then remove it:

docker rm <container_id>

Delete Multiple Containers

docker rm -f <container_id1> <container_id2> <container_id3>

Alternative Method (Windows)

FOR /F "tokens=*" %i IN ('docker ps -aq') DO docker rm -f %i

This command removes all containers.

Listing Containers

View Running Containers

docker ps

ps stands for Process Status.

This command lists only running containers.

View All Containers

docker ps -a

This command lists all containers, including exited containers.

-a stands for all.

View Only Container IDs

docker ps -aq

This command lists all container IDs.

Delete Images

docker rmi -f <image_id>

This command forcefully removes an image.

Naming a Container

To assign a name to a container:

docker run -d --name anyname busybox:1.36

Example:

docker run -d --name my-container busybox:1.36

This creates a container with the name my-container.

Docker – Image and Container Bonding & Client-Server Architecture

Ramya Perumal — Sat, 27 Jun 2026 20:29:32 +0000

Question:

A container is running from an image. If we try to delete the image while the container is running, why can't we delete the image?

Answer:

If we attempt to delete the image, Docker will display an error message stating that the image is being used by a container.

The reason is that each line in a Dockerfile creates a layer.

Each layer is a read-only layer. Once a layer is created, it cannot be modified. If we want to make changes to the image, we need to create a new Dockerfile and build a new image.

When we create a container, only a reference to the image layers is passed to the container. The container has a writable layer, meaning we can create files, modify files, or generate output inside the container. These changes reside only in the container and do not affect the image.

However, the container always depends on the image. That is why we cannot delete the image while the container is using it.

This methodology is called Copy-on-Write (CoW).

Question:

If an image size is 10 GB, what will be the container size?

Answer:

A container contains only references to the image layers. Therefore, the container size consists of the files and changes stored in the writable layer running inside the container.

Question:

What command is used to find the size of a container?

docker ps --size

The Dive application is used to analyze each layer in a Docker image.

Client-Server Architecture

Docker works based on the Client-Server Architecture.

The Docker client sends requests to the Docker daemon, and the Docker daemon responds to the client.

If a requested image is not found on the local system, Docker retrieves it from a public repository.

docker run hello-world

This command pulls the image from the repository and runs it on the local system.

To pull an image from an image repository

docker pull busybox:1.36

To create a container

docker run busybox:1.36

To create a container, execute the `ls` command inside it, and exit immediately

docker run busybox:1.36 ls

To create a container and enter interactive mode

docker run -it busybox:1.36

Interview Questions

Question 1:

In a Client-Server architecture, who sends requests to the server?

Answer: The client.

Question 2:

What happens when a container is created from an image?

Answer: A writable layer is created.

Question 3:

What is the purpose of the writable layer in a container?

Answer: To handle file modifications and store changes made inside the container.

Question 4:

The read-only layers in Docker come from what?

Answer: The base image and its image layers.

Question 5:

What happens to the writable layer when the container is deleted?

Answer: It is discarded.

Question 6:

In a Docker environment, the client interacts with what?

Answer: The Docker daemon.

Question 7:

How do you start a container in interactive mode?

Answer:

docker run -it <image-name>

Docker – Need for Docker and Docker Terminologies

Ramya Perumal — Sat, 27 Jun 2026 20:29:13 +0000

Need for Docker

When more than one application runs on a single physical machine, all the applications have to share the machine's memory, CPU, and computational resources.

Suppose one application consumes more computational power. In that case, the other applications may become slow or even stop responding.

One solution is to run each application on a separate physical server. Although this provides better performance and isolation, the infrastructure cost and maintenance cost become very high.

To overcome this problem, the concept of Virtual Machines (VMs) was introduced.

In a virtual machine environment, each application runs on its own operating system while sharing a single physical machine.

Benefits of Virtual Machines

Reduced hardware cost.
Lower maintenance cost.
Better isolation between applications.
Multiple operating systems can run on a single physical machine.

A software component called a Hypervisor is responsible for virtualizing the physical machine and allowing multiple virtual machines to run on it.

Types of Hypervisors

There are two types of Hypervisors.

Type 1 Hypervisor

A Type 1 Hypervisor is installed directly on the physical machine (bare metal).

Type 2 Hypervisor

A Type 2 Hypervisor is installed on top of a host operating system.

We generally use Type 2 Hypervisors on personal computers. They allocate virtual resources to each virtual machine either manually or dynamically.

However, virtual machines still require a complete operating system for every application, which consumes a significant amount of memory and storage.

This means we are not fully utilizing the operating system resources for every application.

To overcome this limitation, Containers were introduced.

Containers include only the minimum libraries and dependencies required to run an application.

Benefits of Containers

Containers consume much less memory than virtual machines.
Containers start much faster.
Containers are lightweight.
Containers are portable.

An application is packaged as a Docker Image, which can be shared with any number of users and run consistently across different environments.

Docker Terminologies

For better understanding, let's compare Docker concepts with a kitchen.

Docker Concept	Kitchen Analogy
Docker Engine	The kitchen where everything happens.
Dockerfile	The recipe that contains the ingredients and preparation steps.
Docker Image	The finished dish prepared using the recipe.
Docker Container	A serving (portion) of the finished dish.
Docker Registry	A pantry that stores many dishes (images) with different tags.
Docker Daemon	The chef who prepares the dish (image) by following the recipe (Dockerfile).

Installing Docker

Download and install Docker Desktop.

Once the installation is complete, Docker is ready to use.

List Images Available on the Local Machine

docker images

This command lists all Docker images available on the local machine.

Pull an Image from Docker Hub

If the requested image is not available locally, Docker automatically downloads it from the Docker Registry.

docker pull hello-world

If no tag (version) is specified, Docker downloads the latest version by default.

docker pull hello-world:latest

To download a specific tagged version:

docker pull hello-world:nanoserver-ltsc2025

Create a Container from an Image

docker run hello-world:latest

List Containers

docker ps -a

-a displays all containers, including exited containers.

Interview Questions

Question 1

What is a Virtual Machine (VM)?

Answer:

A Virtual Machine is a software emulation of a physical computer. It behaves like a separate computer with its own operating system.

Question 2

What does a Hypervisor do in virtualization?

Answer:

A Hypervisor allows multiple virtual machines to run on a single physical host by managing and allocating hardware resources.

Question 3

What are the advantages of using containers?

Answer:

Containers are lightweight.
Containers start much faster than virtual machines.
Containers are portable and can run consistently across different environments.
Containers package only the required dependencies.

Note:

Containers share the host operating system kernel. If the host kernel encounters a critical issue, all containers may be affected.

Virtual machines have separate operating systems. Therefore, if one virtual machine crashes, the others continue to run independently.

Question 4

Which type of Hypervisor runs directly on physical hardware?

Answer:

Type 1 Hypervisor.

Question 5

What is the difference between a Virtual Machine and a Container?

Answer:

Containers share the host operating system kernel.
Virtual machines run their own operating system on top of a Hypervisor.

Question 6

What is Docker primarily used for?

Answer:

Docker is primarily used to containerize applications, ensuring portability, consistency, and providing only the minimum required dependencies to run the application.