DEV Community: Farhim Ferdous

What is a Server?

Farhim Ferdous — Tue, 24 Jan 2023 14:07:11 +0000

Servers 101: Understanding What It Is, Why It Matters & More

Originally published here.

How are you able to access any website on the Internet? What is the backbone of modern networks that Businesses and Governments rely upon?

Yes, Servers.

But what exactly are they and how do they work?

Keep reading to find out!

Introduction

In this blog, we will learn all about Servers including:

What is a Server?
Why should we learn about Servers?
Why do they exist?
Server Computers VS Desktop Computers
How do they work?
Types of Servers

Let’s begin by answering…

What is a Server?

A Server is a computer program that provides some resources or functionality to other computer programs.

As a figure of speech, the computers running the server programs are often also referred to as servers. Although that’s technically incorrect.

Why should we learn about Servers?

Servers play a vital role in the information age of today. The following are the main benefits of learning about Servers:

HIGH PAYING Jobs: Understanding servers and how they work can open up job opportunities in IT, especially DevOps that pay top dollar!
Cloud Computing: Due to the enormous business value, many companies are moving their data and applications to remote servers offered by AWS, Azure, GCP, etc. A key part of Cloud Computing is - Servers. Thus, mastering them can be a valuable skill for anyone interested in working in the technology industry.
Security: Servers are often the target of cyber attacks, so understanding how to secure them and protect against potential threats is essential.
Improved performance: By understanding how servers work and how to manage them, you can optimize performance and speed up your website, application, or service.
Cost-effectiveness: By understanding servers, you can make better-informed decisions about the hardware and software you use, which can help lower costs.
Automation: Many server management tasks are repetitive and toilsome. Automating tasks such as configuration and deployment can save time and reduce human error.
Scalability: Knowing how to set up and manage servers allows you to easily scale your infrastructure to meet the needs of a growing user base.

Yes, learning about servers is surely advantageous.

But, you may ask…

Why do Servers exist?

Servers exist to provide a centralized location for storing, managing, and distributing resources and information.

The origins of servers can be traced back to the 1960s when mainframe computers were used by large organizations to store and manage data. These early servers were large and expensive, and were primarily used by government agencies and large corporations.

As technology advanced, servers became smaller, more powerful, and more affordable. This led to the widespread adoption of servers in various industries, including business, education, healthcare, and government.

Today, servers form the foundation of modern networks like the Internet, enabling the ability to connect with people and resources worldwide. There is no doubt they play a crucial role in the functioning of businesses, and industries of all kinds. Without servers, the efficient and effective functioning of these systems would not be possible.

Server Computers VS Desktop Computers

Server computers are quite different from desktop computers that we usually use at home or work. Although both types share similarities, they have major differences in terms of their design, capabilities, and intended use.

Server computers are generally packed in data centers for heat and efficiency. They are often virtualized to handle the shared needs of multiple users and projects. These machines are designed to handle heavy workloads and large amounts of requests and data. They typically have high-performance specifications and are built to run 24/7, ensuring high availability and fault-tolerance. Additionally, servers often have redundancy built in, ensuring that if one component fails, another can take its place, minimizing downtime and disruption to users.

Desktop computers, on the other hand, are designed for personal use by a single user and are typically used for tasks such as word processing, internet browsing, gaming, etc. These computers are not built to handle heavy workloads and data like servers computers. Instead, they are intended for more basic, everyday use and are usually only needed for a few hours a day. They come in a variety of forms and specifications, and can be customized to fit a user's specific needs and preferences.

Ok, so…

How do Servers Work?

Servers work by running specialized software programs that allow them to handle multiple requests simultaneously.

‘Servers’ are devices or programs that store and distribute resources. ‘Clients’ are Devices or programs that consume those resources by requesting the servers. Together, this architecture is called the ‘Client-Server Model’.

When a client requests some information or resources from a server, the server processes the request and sends the information or resources back to the client. This forms the ‘Request-Response Cycle’.

Take for example Google Search. When you search for something on Google, your browser as the ‘client’ sends a ‘request’ to Google. When a server from Google gets the request over the Internet, it performs its logic/function and sends your browser a ‘response’, which you see as search results.

Types of Servers

There are numerous types of servers, each with a specific purpose. Some of the most common types include:

Web servers: These servers distribute web pages and documents that are typically consumed by web browsers. Example: Apache, Nginx.
Database servers: These servers store and manage large amounts of data, such as online transactions, inventory, customer information, etc. Example: MySQL, MongoDB.
File servers: These servers store and manage files, such as documents, images, and videos. Example: FileZilla.
Mail servers: These servers manage and distribute email messages.
Print servers: These servers manage and distribute print jobs to connected printers.

And countless other types.

You are probably interacting with a server right now!

Conclusion

Servers have come a long way since the 1960s and have become an essential component of modern networks. Learning about servers can open up job opportunities in fields such as IT and DevOps. It is also becoming increasingly important for businesses and organizations of all sizes.

If you're interested in delving deeper into the world of servers and Cloud Computing, be sure to subscribe to the TechMormo Newsletter for more valuable DevOps content.

Thanks for making it so far!

We have a LOT more exciting DevOps content on the way! 🙌

See you at the next one.

Till then…

Be bold and keep learning.

But most importantly,

Tech care!

Subscribe to TechMormo Newsletter for cutting-edge DevOps content! 🚀

What is DevOps?

Farhim Ferdous — Tue, 24 Jan 2023 13:50:11 +0000

Really understand what DevOps is and the massive opportunities it offers!

Originally published here.
Link to video: https://www.youtube.com/watch?v=cHxMaoe8DL0

What the heck is DevOps?

And why is the demand for DevOps Engineers growing even faster than for Machine Learning or Data Science jobs?

Keep reading to find out…

Introduction

Here’s the agenda for this tutorial:

What is DevOps?
Common DevOps Practices
Core DevOps Technologies
Why learn DevOps?
“Do I HAVE to be a ‘DevOps Engineer’?”
The Challenges in learning DevOps
How TechMormo can help master DevOps

Ok, let’s first answer…

What is DevOps?

DevOps is a concept aimed to deliver better software faster by uniting Software Development (Dev) and Operations (Ops).

The Dev team is responsible for product development, whereas the Ops is responsible for running systems smoothly.

Traditionally these teams have worked separately with conflicting interests.

The Dev team wants to move fast, release new features, and fix bugs.

The Ops team, on the other hand, wants to move slowly and reduce changes to keep things stable.

This conflict of interests used to cause a great deal of tension within organizations, negatively impacting the product.

DevOps breaks down the proverbial walls within organizations so that teams can communicate and collaborate better.

By embracing a set of practices, tools, and cultural changes, DevOps provides faster releases, faster feedback from customers, and ultimately better software.

DevOps Practices

DevOps is a broad and abstract topic. Different companies implement it differently.

But most teams that embrace DevOps usually have a few practices in common:

1) Break down organizational silos to empower cross-functional teams

This enables better communication and collaboration between teams in an organization. It also promotes self-sufficiency enabling teams to develop and deploy their products/services independently. Ultimately, the organization gains unrivaled agility.

2) Release short and incremental changes

Inspired by Agile methods, this practice is key to being quick in an ever-changing industry. Short, incremental changes are faster to develop, review and ship. It also helps to resolve sudden incidents faster when deploying a small change, as opposed to when deploying 10,000 lines of code and multiple breaking changes.

3) Adopt microservice architectures

When you have multiple teams able to own the entire lifecycle of their projects, you naturally get many small loosely-coupled services talking to each other ie. microservices. Although, this is not a necessity, esp. for small companies that can benefit from the simplicity of Monolithic architectures. But when growing or scaling, microservices can help significantly.

4) Version control everything

DevOps Teams often apply version control to many different assets. From design and architectural assets, infrastructure, configuration, and of course - source code. Version control has a very useful property - it creates an immutable view of the history of changes. This enables a team to refer to exact changes made, as well as to roll back easily.

5) Automate everything

By leveraging tools and technologies, DevOps fosters a culture of automation. Tasks that are repeated often, are manual, or are toilsome are usually primary targets of automation. Why automate? It lowers costs, significantly improves speed and throughput, reduces bugs + human error, and lets team members focus on high-value tasks.

DevOps Technologies

By incorporating DevOps practices, a set of tools and technologies emerged.

The following are the core segments of DevOps Technologies:

Scripting languages (eg. Python, Shell Script, Ruby)
Version control system (eg. Git, SVN)
OS & Linux basics (terminal usage, mgmt. of processes, memory, storage, etc.)
Networking & Security basics (IP, Firewalls, DNS, HTTPS, LBs, etc.)
Containerization (eg. Docker, Kubernetes, etc.)
(CI/CD) Continuous Integration / Continuous Delivery (eg. Jenkins, Gitlab-ci, etc.)
Cloud Computing (eg. AWS, Azure, GCP, etc.)
(IaC) Infrastructure as Code (eg. Ansible, Terraform, etc.)
Monitoring & Observability (eg. Elastic Stack, Prometheus, Grafana, etc.)

NOTE: this is neither a fixed nor a comprehensive list. As DevOps practices evolve, so will the technologies associated with them.

Now, learning all these technologies might seem daunting.

Or exciting.

Or both.

Whatever your response is, you might naturally ask…

Why learn DevOps?

Below are the top 3 reasons we love DevOps!

1) Massive learning opportunity

DevOps touches a lot of roles and domains, not just Dev and Ops. This means a DevOps Engineer has to learn continuously, enabling the opportunity to build a diverse skill set.

Depending on the company, this can include elements of Design, Development, Operations, Testing, Management, etc.

It sounds fantastic if you love to learn, but scary if you’re not into learning.

In the coming days of Automation & AI, if you don’t develop a love for learning, you might be in trouble.

Remember, learning is investing in yourself.

As Warren Buffet puts it -

“The best investment you can make is in yourself… The more you learn, the more you’ll earn.”

2) High-impact role

As a DevOps Engineer, you can move (technological) mountains.

Is this a joke?

Nope.

Imagine commanding hundreds of servers at your will, automating manual-repetitive tasks, designing microservices, and having ownership of the entire SDLC.

Your actions will have a direct impact on the business, including how quickly the product can be released, how smoothly your team members can work, etc.

In other words, you will have a highly leveraged role.

Sounds exciting, right?!

Remember, with great power, comes great responsibility.

And big money too! 💸

3) High salary & exploding demand

Companies realize the huge benefits of implementing DevOps. That’s why they are willing to pay top dollar for DevOps Engineers.

For example, an average AWS Solutions Architect makes over $150k/year in the US, according to Glassdoor.

What’s more, the demand is exploding!

In 2021, Bain & Company reported a growth of 443% increase in DevOps job postings from 2015 to 2019. This growth is even higher than postings for Machine Learning (417%), Data Science (167%), or Software Engineering (69%).

Additionally, most Engineers are not competent in DevOps, resulting in low supply and low competition.

This is YOUR opportunity!

You won’t have a hard time finding jobs with DevOps skills.

You want to tap into this huge salary growth. But you think to yourself…

Do I HAVE to be a ‘DevOps Engineer’?

Absolutely not.

Any IT Professional can start learning DevOps, make an impact, and position themselves to be able to earn more.

The shortage of DevOps Engineers means many companies don’t even have a dedicated role for “DevOps”. Instead, existing Software Engineers or System Administrators are usually trained in DevOps skills.

But, most Engineers don’t become competent in DevOps due to…

The Challenges in learning DevOps

Learning DevOps is often overwhelming, whether you’re about to start or already a practitioner.

DevOps is complex: There are a lot of things to learn. Additionally, it is continuously evolving as new technologies come out every day.
Lack of guidance: Due to the complexity and unique backgrounds of individuals, it is difficult to figure out an effective path to DevOps mastery.
Most resources are just theory: Tired of theory lectures and PowerPoint slides that offer no practical value? We are too.
Live classrooms don’t take your pace into account: Ever been stuck in a class that goes too fast or too slow? This often happens for live DevOps training because not everyone learns at the same pace.

How do you master DevOps effectively then?

How TechMormo helps you master DevOps?

At TechMormo, our goal is to help you master DevOps.

Simplicity: we are obsessed with simplifying complex topics to help you become a better DevOps Engineer. We craft lectures with easy-to-understand animations and visuals.
Do projects, start to finish: Learn how to create and scale projects from start to finish, step-by-step. Whether you're just starting out or are already a practitioner, you will find value in our content.
Hands-on practical skills: Don’t just digest Theory. Learn hands-on practical skills that you can implement in your job TODAY.
Learn at your own pace: Stuck in a class that’s too slow or too fast? We create on-demand learning resources that you can use anytime, anywhere.

So, what are you waiting for?

Subscribe to the TechMormo Newsletter to get notified of new cutting-edge DevOps content.

Learn DevOps, stand apart, and reap the rewards! 🏆

Conclusion

In this tutorial, we learnt what DevOps is including its core practices and technologies.

We also discussed why learning DevOps is beneficial, what the challenges are in learning, and how TechMormo can help in your DevOps journey.

DevOps is continuously evolving. The practices and technologies are likely to change over time. But what’s important is to understand its core purpose: “deliver better software faster”.

Thanks for making it so far!

We have a LOT more exciting DevOps content on the way! 🙌

See you at the next one.

Till then…

Be bold and keep learning.

But most importantly,

Tech care!

ভার্চুয়ালাইজেশন কি? (Virtualization) | ডকার কন্টেইনার vs ভার্চুয়াল মেশিন

Farhim Ferdous — Tue, 24 Jan 2023 13:28:26 +0000

ভার্চুয়ালাইজেশনের সহজ বর্ণনা । কন্টেইনার এবং ভার্চুয়াল মেশিন (VM) এর মধ্যে তুলনা

Originally published here.
Link to video: https://youtu.be/K2pBaJeFD7E

ক্লাউড কম্পিউটিং! ডকার! কুবারনেটস! Serverless!…

একজন সফটওয়্যার ইঞ্জিনিয়ার হিসাবে এই সকল শব্দ যদি আপনি না শুনে থাকেন তাহলে আপনি সম্ভবত কোন পাথরের নিচে বসবাস করছেন।

কিন্তু এই সব প্রযুক্তির মধ্যে কোন জিনিসটা কমন আছে?

ভার্চুয়ালাইজেশন!!!

কিন্তু সেটা আসলে কি? এবং এটার সাথে ডকার কন্টেইনার বা ভার্চুয়াল মেশিন এর কি সম্পর্ক?

এই টিউটোরিয়ালটিতে যতটা সম্ভব সহজভাবে আমরা এগুলো বোঝার চেষ্টা করবো।

আমরা একটা উদাহরণ দিয়ে এটা বোঝার চেষ্টা করি।

উদাহরণ - কি সমস্যার সমাধান দরকার?

ধরুন একটি কাল্পনিক কোম্পানি 'XYZ' এর দুটো সফটওয়্যার অ্যাপ্লিকেশান চালাতে হবে: App A ও App B। এখন কোন কারনে একটা অ্যাপ অন্যটার সাথে মানানসই (compatible) নয়।

ধরেন অ্যাপ এ হচ্ছে একটা পাইথন অ্যাপ যেটা লিনাক্সে চালাতে হবে। এবং অন্যটা হচ্ছে একটা C# Dot Net অ্যাপ যেটা Windows এ চালাতে হবে।

দুটো অ্যাপের অপারেটিং সিস্টেম এবং লাইব্রেরি dependency গুলো আলাদা।

এখন কোম্পানি 'XYZ' কি কড়তে পারে?

যদি ভার্চুয়ালাইজেশন বলতে কিছু না থাকে, তখন স্বাভাবিকভাবে কোম্পানি 'XYZ' দুটো কম্পিউটার কিনে আনবে। তারপর দুটো অ্যাপ দুই কম্পিউটার এ সেটআপ করে চালাবে।

চালানর পর হইলো কাহিনী।

লক্ষ্য করুন যে, অ্যাপ A ব্যবহার করছে প্রথম মেশিনের ৩০%, আর অ্যাপ B দ্বিতীয় মেশিনের ৪০%।

আমরা খুব সহজেই এই পদ্ধতির অসুবিধা দেখতে পাই:

প্রতিটি অ্যাপই তার কম্পিউটার পুরোপুরি ব্যবহার করছে না - resource অপচয়

অ্যাপ এর resource (CPU/Ram, etc) কত লাগবে তা অনুমান করা কঠিন - হয় বেশি বা কম অনুমান করা হবে

প্রতিটি নতুন অ্যাপ্লিকেশনের জন্য অতিরিক্ত হার্ডওয়্যার কিনতে হবে - ব্যয়বহুল

নিরাপত্তা ও ম্যানেজমেন্ট প্রতিটি সিস্টেমের জন্য আলাদাভাবে কড়তে হবে - ম্যানেজ করার পেড়া

এই সমস্যাগুলার যদি কোন সমাধান না থাকতো, তাহলে এগুলো খুব একটা বড় বিষয় মনে হত না।

সৌভাগ্যবশত, আমাদের কাছে এই সমস্যাগুলার সমাধান আছে - এবং এটিকে বলা হয়...

ভার্চুয়ালাইজেশন।

ভার্চুয়ালাইজেশন এর সংজ্ঞা (Definition)

ভার্চুয়ালাইজেশন হল সেই প্রযুক্তি যা দিয়ে আমরা একটা physical resource কে ভাগ করে একাধিক virtual resource এ পরিণত করতে পারি।

যেমন, আমরা একটা কম্পিউটারের হার্ডওয়্যার অংশগুলো - যেমন CPU, RAM, Storage, ইত্যাদি এগুলো ভাগ করে অনেকগুলো ভার্চুয়াল ছোট কম্পিউটার বানাতে পারি।

এটা কি কোন গাজাখরি গল্প বললাম?

মোটেও না।

আপনি যদি ভার্চুয়াল মেশিন বা ডকার কন্টেইনার ব্যবহার করে থাকেন, তাহলে এটার ব্যাপারে আপনার ইতিমধ্যে কিছুটা জানার কথা।

ভার্চুয়ালাইজেশনের সাহায্যে, একটি ফিজিক্যাল মেশিনকে কয়েকটি ছোট ভাগে ভাগ করা যায়। প্রত্যেকটি ছোট ভাগ নিজেকে মনে করে সে একজন রিয়েল ফিজিক্যাল মেশিন। কিন্তু সে আসলে জানেনা সে স্বপ্নের ভিতরে - তাই তাকে "ভার্চুয়াল" বলা হয়।

ভার্চুয়াল এনভায়রনমেন্টের ভেতরে যেই অ্যাপগুল চলবে, সেই অ্যাপগুলোর শুধু তার ভার্চুয়াল এনভায়রনমেন্টের নির্ধারিত resource এর অ্যাক্সেস থাকবে।

যদি অ্যাপ A ও অ্যাপ B চালানর পর মেশিনর আরো অতিরিক্ত resource থাকে, তাহলে কোম্পানি XYZ চাইলে আরো একটা অ্যাপ হয়তো চালাতে পারে। (যদিও দীর্ঘ সময়ের জন্য কোন কম্পিউটার ফুল লোডে ব্যবহার করা বুদ্ধিমানের কাজ হবে না)

ভার্চুয়ালাইজেশন থাকায় কি লাভ হোল?

ভার্চুয়ালাইজেশন এর উপকারিতা (Benefits)

হার্ডওয়্যারের খরচ সঞ্চয়: একটা মেশিনর physical resource গুলো (CPU, RAM, etc.) আরো দক্ষতার সাথে ব্যবহার করে আমরা হার্ডওয়্যার খরচ কমাতে পেরেছি। এতেকরে একটি মেশিনে একাধিক প্রোজেক্ট বা এনভায়রনমেন্ট চালানো যাবে।

অ্যাপ্লিকেশন বিচ্ছিন্নতা (isolation): যেহেতু প্রতিটি ভার্চুয়াল এনভায়রনমেন্ট একটি ছোট 'রিয়েল' মেশিনের মত ভান করে, এতেকরে এক ভার্চুয়াল এনভায়রনমেন্টের অ্যাপগুলো অন্য ভার্চুয়াল এনভায়রনমেন্টের অ্যাপের কোন প্রবাভ ফেলে না। তাই, আমরা সহজেই প্রতিটি অ্যাপ আলাদাভাবে develop ও deploy কড়তে পারি।

আর ভালো scalability ও availability: নতুন হার্ডওয়্যার কেনার থেকে ভার্চুয়াল এনভায়রনমেন্ট তৈরি করা বা রিমুভ করা আরো সহজ এবং দ্রুত। ভার্চুয়ালাইজেশনের মাধ্যমে আমাদের অ্যাপগুলো আমরা আরো সহজে scale কড়তে পারব - অর্থাৎ, নতুন মেশিন এড বা রিমুভ করা। আর, অ্যাপের একাধিক কপি চালিয়ে আমরা availability বৃদ্ধি কড়তে পারবো - অর্থাৎ, অ্যাপটা যে চলছে সেটার সম্ভাবনা বাড়াতে পারবো।

আরও সহজ ম্যানেজমেন্ট ও নিরাপত্তা: যেহেতু অ্যাপ্লিকেশনগুলো তাদের ব্যবহার করা ফিসিকাল মেশিনগুলোর থেকে বিচ্ছিন্ন (ভার্চুয়ালাইজেশনের দ্বারা), তাই ফিসিকাল মেশিনগুলোকে কেন্দ্রীয়ভাবে পরিচালিত করা যেতে পারে এবং নিরাপত্তা নীতিগুলি (security policies) কেন্দ্রীয়ভাবে প্রয়োগ করা যেতে পারে। একশটা মেশিনে ঢুকে প্রত্যেকটা আপডেট করা, নাকি এক জায়গা থেকে একশোটা মেশিন আপডেট করা - কোনটা বেছে নিবেন?

ভার্চুয়ালাইজেশনের প্রকারভেদ (Types)

ভার্চুয়ালাইজেশনের বিভিন্ন প্রকার রয়েছে, যেমন:

হার্ডওয়্যার (Hardware) ভার্চুয়ালাইজেশন - ভিএম (VM)

অপারেটিং সিস্টেম (OS) ভার্চুয়ালাইজেশন - কন্টেইনার (Containers)

নেটওয়ার্ক (Network) ভার্চুয়ালাইজেশন

ডেটা (Data) ভার্চুয়ালাইজেশন

ইত্যাদি।

কিন্তু আমরা শুধুমাত্র প্রথম দুইটিতে ফোকাস করবো এখন।

তুলনা: কন্টেইনার, ভার্চুয়াল মেশিন ও বেয়ার মেটাল

বেয়ার মেটাল (Bare metal)

এটি প্রথম পদ্ধতি যা কোম্পানি XYZ নিয়েছে।

কম্পিউটার আসার পর প্রথম প্রথম একটি অ্যাপ্লিকেশন চালানোর উপায় ছিল সরাসরি ডেডিকেটেড হার্ডওয়্যারের উপর। অর্থাৎ - একটি মেশিন, একটি অপারেটিং সিস্টেম (OS), একটি অ্যাপ।

বেয়ার-মেটালের সবচেয়ে বড় সুবিধা হল, হোস্ট, লাইব্রেরি এবং অ্যাপের মধ্যে ভার্চুয়ালাইজেশন লেয়ার না থাকায় এটি সেরা পারফরম্যান্স দেয়। যদিও ভার্চুয়ালাইজেশনের পেনাল্টি সাধারণত বেশিরভাগ ক্ষেত্রে খুব গুরুতর নয়।

যখন কেউ কাজ করার জন্য একটি কোম্পানির আইটি স্ট্যাক বেছে নিতো, তখন তারা সাধারণত সেই কোম্পানির নির্দিষ্ট হার্ডওয়্যার, ওএস বা লাইসেন্স চুক্তিতে লক-ইন হয়ে জেত। এতে করে - একক মেশিনে একাধিক অ্যাপ্লিকেশন চালানো খুব কঠিন হত।

ফলে, কম্পিউটারগুলোর resources অপচয় হতো, বিভিন্ন টিমের/অ্যাপের জন্য নতুন এনভায়রনমেন্ট সহজে বানানো জেত না, এবং scale করা কঠিন ছিল, কারণ প্রতিটা নতুন অ্যাপের জন্য নতুন মেশিন কেনাটা ব্যয়বহুল।

সমাধান কি তাহলে?

ভার্চুয়াল মেশিন (VM)

১৯৯০ এর দশকে ভার্চুয়াল মেশিনের ব্যবহার বৃদ্ধি পেয়ে ভার্চুয়ালাইজেশন ব্যাপক আকারে ছড়িয়ে পড়লো।

ভার্চুয়াল মেশিন বানানোর মুল প্রযুক্তি হোল হাইপারভাইজার (hypervisor)।

একটি হাইপারভাইজার হল এমন একটি সফ্টওয়্যার যা কম্পিউটার হার্ডওয়্যারের একটি নির্দিষ্ট অংশ বা সম্পূর্ণ কম্পিউটারকে অনুকরণ (emulation) করে। এতে করে ফিসিকাল কম্পিউটারটাকে ভাগ করে একাধিক ভার্চুয়াল কম্পিউটার বানানো জায় - যেগুলোকে ভার্চুয়াল মেশিন বলা হয়।

যে কম্পিউটারটি হাইপারভাইজার চালায় তাকে হোস্ট (host) সিস্টেম বলা হয়, এবং হাইপারভাইজার দ্বারা তৈরি ও পরিচালিত ভিএমগুলিকে গেস্ট (guest) সিস্টেম বলা হয়।

হাইপারভাইজাররা সরাসরি হার্ডওয়্যারের উপরে (টাইপ-1) অথবা একটি ওএসের উপরে (টাইপ-2) বসতে পারে।

ভার্চুয়াল মেশিন দারুণ এক প্রযুক্তি, কারণ তা দিয়ে আমরা বেয়ার মেটালের অনেক সমস্যার সমাধান করেছি।

কিন্তু ভার্চুয়াল মেশিনেরও কিছু ত্রুটি রয়েছে।

যেমন, প্রতিটি ভিএম একটি সম্পূর্ণ অপারেটিং সিস্টেম এবং এর প্রয়োজনীয় হার্ডওয়্যার ভার্চুয়ালাইজ করে। এটা কিছু ক্ষেত্রে আসলেই কার্যকর, যেমন জখন আমরা সুরক্ষার জন্য উচ্চ বিচ্ছিন্নতা (isloation) চাই।

তবে অনেক ক্ষেত্রেই তা অপ্রয়োজনীয়।

ভিএম থেকে হালকা কোন ভার্চুয়ালাইজেশন আছে কি?

কন্টেইনার (Container)

কনটেইনারগুলি VM-এর মতো সম্পূর্ণ ফিজিক্যাল মেশিনকে ভার্চুয়ালাইজ না করে শুধুমাত্র অপারেটিং সিস্টেমকে ভার্চুয়ালাইজ করে।

তাদের একটি গেস্ট ওএস এবং হাইপারভাইজারের প্রয়োজন নেই।

পরিবর্তে, একটি হোস্ট মেশিনে চলমান সমস্ত কন্টেইনার হোস্ট সিস্টেমের ওএস কার্নেল (OS Kernel) ভাগ করে। শুধুমাত্র অ্যাপ্লিকেশন কোড, তাদের লাইব্রেরি এবং dependency গুলো কন্টেইনার ধারণ করে।

তাই কন্টেইনার অত্যন্ত হালকা। এবং ভিএম বানানোর থেকে কন্টেইনার বানানো অনেক দ্রুত!

এছাড়াও, কন্টেইনার ভিএম থেকে বেশি portable. জেহুতু কন্টেইনার ফিজিক্যাল মেশিনের হার্ডওয়্যার ভার্চুয়ালাইজ করেনা, সেজন্যে কন্টেইনারগুলোকে বিভিন্ন cloud প্লাটফর্ম বা ডেটা সেন্টেরে সহজে চালানো সম্ভব।

এই কারণে কন্টেইনার দ্বারা অনেক DevOps সল্যুশন এবং বিষয় বৃদ্ধি পেয়েছে - যেমন মাইক্রোসার্ভিসেস, CI/CD পাইপলাইন, horizontal scaling, ইত্যাদি।

কিন্তু কন্টেইনারই সব সমস্যার সমাধান নয়।

যেহেতু অনেক কন্টেইনার একই হোস্ট মেশিনে চলে এবং হার্ডওয়্যার ভার্চুয়ালাইজ করে না, তাই তারা VM-এর তুলনায় কম বিচ্ছিন্নতা (isolation) প্রদান করে।

এটি কিছু নিরাপত্তা বা সম্মতি (compliance) সংক্রান্ত সমস্যার জন্ম দিতে পারে, যা কিছু কিছু প্রকল্পে/প্রোজেক্টে একটি গুরুত্বপূর্ণ বিবেচনা হতে পারে।

কিন্তু এটি বেশিরভাগ অ্যাপ্লিকেশনের জন্য চিন্তার বিষয় নয়। So relax!

উপসংহার (Conclusion)

বেয়ার মেটালের পেড়া থেকে আমরা ভার্চুয়ালাইজেশনের বিপ্লব পেয়েছি।

আজকের বেশিরভাগ অ্যাপ্লিকেশনগুলি ভার্চুয়ালাইজেশনের দ্বারা চলে। এমনকি ক্লাউড কম্পিউটিং ও এটার দ্বারা বাস্তবায়ন করা।

OS ভার্চুয়ালাইজেশনের ধারণাটি প্রায় ২০০০ দশকের প্রথম থেকেই প্রচারিত। কিন্তু আধুনিক কন্টেইনার যুগ ২০১৩ সাল থেকে শুরু হয়, ডকারের হাত ধরে।

TechMormo তে আমাদের ডকার কন্টেইনার খুব প্রিয়! আমাদের ইতিমধ্যে ডকার নিয়ে বেশ কিছু ইংলিশ ব্লগ ও ভিডিও বানানো হয়েছে।

Docker Made Easy

Networking in Docker

Blogs on Docker

আপনি কি বাংলাতে ডকার শিখতে চান? নাকি অন্য কোন DevOps বিষয় শিখতে চান?

কমেন্ট করে বা Feedback পেইজ এ আমাদের জানবেন প্লিস।

আজ এই পর্যন্তই।

Be bold and keep learning.

But most importantly,

Tech care!

Subscribe to TechMormo Newsletter for cutting-edge DevOps content! 🚀

What is Docker Compose? (with demo)

Farhim Ferdous — Sun, 30 Oct 2022 20:11:34 +0000

Learn what Docker Compose is, how it makes the life of an Engineer easier and possible use cases

Link to video: https://youtu.be/yFvO8Atszl8

What the heck is Docker Compose and how does it make the life of an Engineer like you easier?

This tutorial will answer that (and more), as simply as possible.

Introduction

Agenda for this tutorial:

What is Docker Compose?
Why it exists? How does it differ from docker run?
How to use Docker Compose?
When to use it? - use cases

Absolutely new to Docker? Check out: What is Docker?

Want to learn more about Docker Networking? Check out: Docker Networking Summary.

Alright…

What is Docker Compose?

Docker Compose is a tool for defining and running multi-container applications.

With Compose, you use a YAML file to configure your application’s services (containers). Then, with a single command, you can build, start or delete your application services.

Why Docker Compose exists? How does it differ from `docker run`?

Running multiple containers is a very common scenario.

Take for example a WordPress (WP) application. It consists of a WordPress service that talks to a MySQL database.

We could run both containers using two docker run commands with a bunch of cli arguments. The db container might be started like this:

docker run -d \
  --name db \
  --restart always \
  -v db_data:/var/lib/mysql \
  -e MYSQL_ROOT_PASSWORD=supersecret \
  -e MYSQL_DATABASE=exampledb \
  -e MYSQL_USER=exampleuser \
  -e MYSQL_PASSWORD=examplepass \
  mysql:5.7

Additionally, we might need to isolate these containers from the host or other containerized applications. We could create or remove networks with docker network commands, and modify docker run to take the network as an argument.

Typing out these verbose commands might be fine once or twice. But as the number of containers and configurations grows, they become increasingly harder to manage.

With Compose, we simply define the application’s configuration on a YAML file (named docker-compose.yml by default) like this:

version: '3.9'

services:
  db:
    image: mysql:5.7
    restart: always
    environment:
      MYSQL_DATABASE: exampledb
      MYSQL_USER: exampleuser
      MYSQL_PASSWORD: examplepass
      MYSQL_ROOT_PASSWORD: supersecret
    volumes:
      - db_data:/var/lib/mysql
  wordpress:
    image: wordpress
    restart: always
    ports:
      - 8080:80
    environment:
      WORDPRESS_DB_HOST: db
      WORDPRESS_DB_USER: exampleuser
      WORDPRESS_DB_PASSWORD: examplepass
      WORDPRESS_DB_NAME: exampledb
    volumes:
      - wordpress_data:/var/www/html

volumes:
  wordpress_data:
  db_data:

This file defines 2 services, db and wordpress. It also specifies the configuration options for each - like the image, environment variables, published ports, volumes, etc.

NOTE: if you don’t understand all these options yet, that’s fine.

After creating this file, we execute docker compose up, and Docker builds and runs our entire application in a new isolated environment (bridge network by default).

Similarly, we can use the docker compose down command to tear everything down (except volumes).

Easy! Right? 👌

This is how compose simplifies running multi-container applications on a single host.

Being able to declare and reuse the configuration as a file makes the life of an Engineer much easier. It also allows us to version control, run tests and review our configuration just like the application’s source code.

Besides the benefits above, Compose provides the following key features:

Have multiple isolated environments on a single host
Preserve volume data when containers are created
Only recreate containers that have changed
Share variables or configurations between environments

That’s all good to know. But let’s learn…

How to use Docker Compose?

Source code for this demo: https://github.com/AluBhorta/docker-compose-demo

Step 1: Install Docker & Docker Compose

First of all, make sure you have installed:

NOTE: since Compose version 2, we use docker compose command instead of docker-compose. This tutorial uses version 2.

Step 2: Create a sample web application

Open up a terminal, create a new directory and switch into it:

mkdir docker-compose-demo
cd docker-compose-demo

Add the code for a simple Python web app on a file named app.py:

import time

import redis
from flask import Flask

app = Flask(__name__)
cache = redis.Redis(host='redis', port=6379)

@app.route('/')
def hello():
    count = cache.incr('hits')
    return 'Hello World! I have been seen {} times.\n'.format(count)

This creates a Flask app with a single HTTP endpoint (/). This endpoint returns how many times it has been visited. The count is stored and incremented as an integer with a key named hits in a Redis host named redis.

Then we add the Python dependencies to a requirements.txt file:

flask
redis

After that, we create a Dockerfile - to create a Docker image based on this application:

FROM python:3.7-alpine

WORKDIR /code

RUN apk add --no-cache gcc musl-dev linux-headers

COPY requirements.txt requirements.txt
RUN pip install -r requirements.txt

EXPOSE 5000

COPY . .

ENV FLASK_APP=app.py
ENV FLASK_RUN_HOST=0.0.0.0

CMD ["flask", "run"]

This tells Docker to:

Build an image starting with the Python 3.7 Alpine Linux image
Set the working directory to /code
Install gcc and other dependencies with the apk package manager
Copy requirements.txt from host to image
Install the Python dependencies with pip
Add metadata to the image to describe that the container is listening on port 5000
Copy the current directory . in the project to the workdir . in the image
Set environment variables used by the flask command
Set the default command for the container to flask run

Then we create a docker-compose.yml file for us to use Docker Compose:

version: "3.9"

services:
  redis:
    image: "redis:alpine"
  web:
    build: .
    ports:
      - "8000:5000"
    depends_on:
      - redis

This Compose file defines two services: web and redis.

The redis service uses a public redis:alpine image pulled from the Docker Hub registry.

The web service uses an image that’s built from the Dockerfile in the current directory (.). It then maps port 8000 on the host to port 5000 on the container where the flask server will be running. It also specifies that web depends on redis so that Docker knows to start redis before web.

NOTE: since Compose creates a new bridge network on project startup, web can reach redis simply by using the service’s name ie. redis.

Step 3: Run and test the application

To run the application, all we have to do now is:

docker compose up

Docker will automatically pull the redis image, build our web image and start the containers.

Once deployed, we should now be able to reach the application at localhost:8000 on your browser.

Or alternatively, use curl on a separate terminal to reach the flask application like so:

curl localhost:8000

You should see something like this:

Hello World! I have been seen 1 times.

The count is incremented every time we make a request.

Awesome! 👌

We can list the containers of the Compose project with:

docker compose ps

NOTE: docker compose up will by default attach to your terminal and print the logs from the services. We can use ctrl+c to detach that terminal, but it will stop the services.

To run the services in the background, use -d the flag:

docker compose up -d

If you want to view the logs, use:

docker compose logs -f

NOTE: -f will follow the log output as new logs are generated.

Step 4: Modify the application

Changes are inevitable.

Let us make a change to our app.

The web container is printing out a warning:

WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead.

In production, we definitely want to run a production-grade server like gunicorn instead of the development server we get with flask run.

So, let’s first add gunicorn to requirements.txt:

flask
redis
gunicorn

Then remove the last 3 instructions of Dockerfile and add a new CMD instruction:

FROM python:3.7-alpine

WORKDIR /code

RUN apk add --no-cache gcc musl-dev linux-headers

COPY requirements.txt requirements.txt
RUN pip install -r requirements.txt

EXPOSE 5000

COPY . .

CMD ["gunicorn", "--bind", "0.0.0.0:5000", "app:app"]

Images built from this Dockerfile will now run gunicorn instead of the flask dev server.

Once we’ve made the changes, we need to rebuild the web image:

docker compose build

Then we restart the web service to use the new image:

docker compose up web -d --no-deps -t 1

NOTE:

--no-deps tells Docker not to (re)start dependent services.
-t 1 specifies the container shutdown timeout of 1 second instead of the default 10s.

Try to reach web again:

curl localhost:8000

It should work as expected.

But if you check the logs:

docker compose logs web -f

You will notice we don’t have the warning anymore since we’re using gunicorn.

Noice!

Step 5: Clean up

To remove all the services, we simply run:

docker compose down

If we had defined volumes in the services (like in the WordPress example), the volumes wouldn’t be automatically removed with docker compose down. This is mainly to avoid accidental deletion of data.

To tear down everything including volumes, use the -v flag:

docker compose down -v

Congratulations! You can now Compose! 🙌

When to use Docker Compose? - use cases

*Development environments*

When developing software, the ability to run applications and their dependencies in isolated environments is crucial. For example, you might have a dependency on another team’s application, which in turn might have its own set of complexities like configuring the database in a particular way. By using Compose, you can run the whole stack or remove it with a single command.
*Automated testing environments*

Automated workflows like CI/CD pipelines generally require tools to easily create and destroy environments. Containers are ideal for such environments due to their low resource footprint and speed. By using a configuration file, Compose provides a convenient way to create and destroy such environments for your test suite.
*Single host deployments*

Although Compose was mainly developed for development and testing workflows, it is sometimes used in production for running containers on a single host. While Compose is improving, it is not an orchestrator like Swarm or Kubernetes, but more of a wrapper around Docker’s API. Check out the official documentation before using compose in production.

Conclusion

In this tutorial, we learnt what Docker Compose is, why it exists, how to use it and when to use it.

By specifying container configuration in a file, Compose simplifies running multi-container workloads on a single host.

If you found this tutorial helpful, leaving a like, comment or subscribing will really help us out.

You might also enjoy our YouTube Channel.

Thanks for making it so far!

We have a LOT more exciting DevOps content on the way! 🙌

See you at the next one.

Till then…

Be bold and keep learning.

But most importantly,

Tech care!

Docker Networking Summary | Networking in Docker #8

Farhim Ferdous — Mon, 24 Oct 2022 11:39:16 +0000

Summary of Docker Networking Series

Link to video: https://www.youtube.com/watch?v=3c0miSU038E

How do Docker Containers communicate using networks?

If you have gone through all the tutorials in this series, you should know that well by now.

Introduction

This blog is the 8th and last tutorial in a series for Docker Networking.

If you are looking to learn more about the basics of Docker, I’ll recommend checking out the Docker Made Easy series.

Summary

In this series, first we answered “Why is networking important in Docker?”.

Then we learnt about some core networking concepts like the Internet Protocol (IP), network interfaces, subnets, network ports, DNS, etc.

Then we discussed about Docker Network Drivers, which is how Docker hides the network details and simplifies how to attach different types of networks to containers.

We then dived into the of the 4 main Network Drivers:

none driver - isolates a container by disabling its network.
host driver - allows the container to share the network stack of the host system.
bridge driver - creates an internal network within a single host.
overlay driver - creates a distributed network able to span multiple hosts.

For each of the Network Drivers, we learnt how to use it, some possible use cases and their limitations.

Conclusion & what’s next?

Thank you so much for making it so far! 🎉 🫂

It has been a great pleasure and learning experience to share about Docker Networking with you. I myself got to learn many important concepts and details in the making of this series.

What do you want to learn next?

Want more Docker? What about Kubernetes, AWS, Linux, Git, Python, or something else entirely?

We need YOUR FEEDBACK 🎯.

This is just the start! We’re actively working to simplify complex DevOps + Engineering topics, and help you become a more competent Engineer.

If you found any mistakes or have any suggestions to improve our content, please let us know.

See you on the next one.

Till then…

Be bold and keep learning.

But most importantly,

Tech care!

The Overlay Network Driver | Networking in Docker #7

Farhim Ferdous — Wed, 12 Oct 2022 13:21:48 +0000

Learn how the Overlay driver simplifies multi-host networking on Docker, how to use it, possible use cases and limitations

Link to video: https://www.youtube.com/watch?v=gJVcKVdYhpI

What is the overlay network driver in Docker and how does it power docker swarm and multi-host container networking?

This blog will try to answer that (and more) as simply as possible.

Introduction

This blog is the seventh one in a series for Docker Networking.

If you are looking to learn more about the basics of Docker, I’ll recommend checking out the Docker Made Easy series.

Here’s the agenda for this blog:

What is the overlay network driver?
How to use it?
When to use it? - possible use cases
What are its limitations?

Here are some quick reminders:

A Docker host is a physical or virtual machine that runs the Docker daemon.
Docker Network drivers enable us to easily use multiple types of networks for containers and hide the complexity required for network implementations.

Alright, so...

What is the `overlay` driver?

The overlay network driver creates a distributed network that can span multiple docker hosts.

Overlay networks were designed to be used with docker swarm services.

Docker swarm is a container orchestration tool (like Kubernetes) that allows us to manage multiple containers on multiple hosts.

To deploy an application (ie. docker image) on the swarm, we create a swarm service, which can run 1 or more containers from the image.

They will make more sense when we actually try them out in a moment.

When a docker host initiates or joins a swarm, 2 new networks are created:

a default overlay network called ingress which handles the control and data traffic related to swarm services.
and a virtual bridge network called docker_gwbridge that connects overlay networks to the individual Docker daemon’s physical network.

Like custom bridge networks we saw in the last tutorial, custom or user-defined overlay networks can be created using the docker network create command (using the --driver overlay option).

Let’s directly see overlay drivers in action…

How to use the `overlay` driver?

I encourage you to follow along to get the most out of this tutorial.

Initialize the swarm

Provision hosts with network connectivity

To see the full potential of overlay networks, we need multiple hosts. Let’s go with 3.

The simplest solution is to use a Virtual Machine tool like VirtualBox or UTM (on Mac) to create multiple VMs.

For multiple machines to communicate together as a swarm using overlay, docker requires the following ports to be open between the hosts:

TCP port 2377 for cluster management communications
TCP and UDP port 7946 for communication among nodes
UDP port 4789 for overlay network traffic

If you’re running VMs locally, make sure the VMs can reach each other over the network (ie. select ”Internal Network” on VirtualBox, or “Shared Network” on UTM).

If you’re running VMs on the cloud or in a strict networked environment, make sure to update your firewall rules / security groups to account for the open port requirements.

Before moving forward, ensure connectivity between your hosts with something like ping.

I have created 3 Debain Linux VMs named node1, node2 and node3.

Install Docker

Follow the official docker instructions to install Docker Server on all your hosts.

https://docs.docker.com/engine/install/

Create the swarm and join the nodes

On the first node, we will run the following:

docker swarm init

This will initialize the swarm with the first node as the manager and the only member for now.

It should also print the command that will let us add other nodes into the swarm as workers, like the following:

docker swarm join --token SWMTKN-1-07ti6ugkt6hia44ljws48eik9t4v0806515jkodf3zn1pb7uod-15c75qxan0guvqmcswt4blz2l 192.168.64.4:2377

We will copy the full docker swarm join command and run it on each of our remaining nodes.

When all are successfully joined, we can check the nodes by running the following on the manager node:

docker node ls

We can also check that the 2 new networks were created with:

docker network ls

Awesome, we now have a docker swarm cluster with an overlay network!

The default overlay network - `ingress` and routing mesh

Let us start off by creating a service named whoami using the containous/whoami image:

docker service create \
    --name whoami \
    -p 8000:80 \
    containous/whoami

This image runs a tiny webserver on port 80 inside the container, and simply returns some OS and HTTP information.

The -p 8000:80 option tells docker to map port 80 inside the container to port 8000 on the host.

But on which host?

Check which host (NODE) the container is running on using:

docker service ps whoami

Let’s say the container is running on node2.

How can we confirm that the ingress network is being used?

Switch to the node2 terminal and get the container ID:

CONTAINER_ID=$(docker ps --filter name=whoami.1 --format '{{.ID}}')

Then inspect its network:

docker inspect $CONTAINER_ID --format='{{json .NetworkSettings.Networks}}' | jq .

You should see a json object named ingress, confirming our hypothesis.

Since the service has been published on port 8000, if we send a request with curl from node2, like:

curl 127.0.0.1:8000

We will get something like this:

Hostname: fb772a20940c
IP: 127.0.0.1
IP: 10.0.0.28
IP: 172.18.0.3
RemoteAddr: 10.0.0.3:59882
GET / HTTP/1.1
Host: 127.0.0.1:8000
User-Agent: curl/7.64.0
Accept: */*

Not surprising.

But the cool thing is, if you try to run the same curl command on the other hosts, you will still get the same response (except for RemoteAddr), even though those nodes don’t have the container running in them. Try it out!

Why is that?

All swarm nodes participate in an ingress routing mesh. The routing mesh enables each node in the swarm to accept connections on published ports for any service running in the swarm, even if there’s no container running on the node.

This means, whenever a docker swarm service publishes a port (with -p/--publish), that service is reachable on that port across all swarm nodes. The overlay network is responsible for routing the traffic to the correct container on the correct node.

But what if there are more than 1 containers?

Let’s find out!

Running multiple containers of a service is as simple as:

docker service scale whoami=3

In a few seconds, we should have 3 replicas of the whoami container running.

Check which host (NODE) the containers are running on using:

docker service ps whoami

We should see that all 3 nodes are assigned a container.

Docker swarm is smart enough to distribute the container workload among the available nodes.

If we now curl again on any of the nodes with:

curl 127.0.0.1:8000

We should see that the Hostnames and IPs are also changing.

This is because, containers running under a swarm service on the overlay network are automatically load-balanced!

Awesome.

Now, what if we want to attach a standalone container (not a swarm service) on the overlay network, for example to communicate across hosts?

If we try to do the following:

docker run -d --rm \
    --network ingress \
    --name app1 \
    nginx:alpine

This will raise an error like “PermissionDenied desc = network ingress not manually attachable.”

The default overlay network (ingress) doesn’t allow attaching standalone containers. For that, we can use…

User-defined Overlay Networks

At first, we have to create a network with the overlay driver option:

docker network create \
    --driver overlay \
    --attachable \
    my-overlay

By using the --attachable flag, docker will allow us to attach standalone containers to it.

If we try to run a standalone container again but this time using our custom overlay network my-overlay:

docker run -d --rm \
    --network my-overlay \
    --name app1 \
    nginx:alpine

it should work.

Since app1 is lonely, let’s run a service with 3 replicas (containers) on my-overlay as well:

docker service create \
    --name web \
    --network my-overlay \
    --replicas 3 \
    nginx:alpine

NOTE: we have not published any ports, but this image runs nginx server on port 80 inside the container.

Now, web service containers can communicate with app1 container and vice versa!

Really?

Try to reach web service from app1 simply by using the name of the service with curl:

docker exec app1 curl web

It should print out the index.html page of nginx.

To test the other way around ie. communication from web service to app1, first, get the container id of a web service container.

CONTAINER_ID=$(docker ps --filter name=web --format '{{.ID}}' | head -n 1)

Then use exec again to curl app1:

docker exec $CONTAINER_ID curl app1

See the nginx index page?

Congratulations! You now know how to use overlay networks! 🎉

Of course, there’s a lot more to it but we have covered the essentials. Feel free to learn more from the official docs.

Clean up

Let’s clean up our mess.

Remove app1:

docker rm -f app1

Then remove the services from the manager node:

docker service rm whoami web

After the services are down, remove the custom overlay:

docker network rm my-overlay

Then, remove each of the worker nodes from the swarm (by running on each worker node):

docker swarm leave

Finally, make the manager leave the swarm:

docker swarm leave --force

Well done! 👏

When to use the `overlay` driver? - possible use cases

If you have multiple nodes to deploy your containerized workloads, but you have a small team with limited containerization experience.
If you need a simple container orchestrator like docker swarm, but do not have the time/resources to invest in more complex orchestrators like Kubernetes.

Limitations of the `overlay` driver

overlay driver is slower than host driver: If you need the highest network performance and your workload is running on a single host, or you are okay with setting up the OS level routing by yourself, then the host driver would be more suitable than overlay. This is because host driver directly uses the host’s network, instead of having to setup separate interfaces, and port-mapping configurations.
Docker (swarm) does not natively support autoscaling: autoscaling is not supported out of the box as you need to provision new infrastructure/servers which is out of the scope of docker. There are 3rd party tools/scripts that can help, but if you really need the ability to autoscale, you should consider something like Kubernetes or AWS ECS.

Conclusion

In this blog, we learnt about the overlay network driver in docker - what it is, how to use it, and some possible use cases and limitations.

By connecting multiple docker daemons in a swarm, the overlay driver simplifies multi-host networking. But its drawbacks are to be kept in mind, especially when using it in production.

I hope I could make things clearer for you, be it just a tiny bit.

In the next blog, we will summarize what we learnt in this docker networking series.

Thanks for making it so far! 👏

See you at the next one.

Till then…

Be bold and keep learning.

But most importantly,

Tech care!

The Bridge Network Driver | Networking in Docker #6

Farhim Ferdous — Wed, 15 Jun 2022 14:10:12 +0000

Learn what the Bridge driver is in Docker, how it simplifies multi-container networking on a single host, how to use it, possible use cases and limitations

Link to video: https://www.youtube.com/watch?v=Z5QLPkp-Ogo

What is the bridge network driver in Docker and how does it simplify networking on a single host?

This blog will try to answer that (and more) as simply as possible.

Introduction

This blog is the sixth one in a series for Docker Networking.

If you are looking to learn more about the basics of Docker, I’ll recommend checking out the Docker Made Easy series.

Here’s the agenda for this blog:

What is the bridge network driver?
How to use it?
When to use it? - possible use cases
its limitations

Here are some quick reminders:

A Docker host is a physical or virtual machine that runs the Docker daemon.
Docker Network drivers enable us to easily use multiple types of networks for containers and hide the complexity required for network implementations.

Alright, so...

What is the `bridge` driver?

A bridge driver can be used to create an internal network within a single Docker host.

A network created using the bridge driver forms a namespace that is separate from the host's network namespace. Therefore, containers running on bridge networks are isolated from the host's network.

This driver is used most often for applications that require one or more containers running on a single host.

When Docker is started, it automatically creates a default bridge network named bridge, and uses it for containers that do not specify any networks explicitly. We can also create many custom ‘user-defined’ bridge networks.

User-defined bridge networks are superior to the default bridge network in the following ways…

Differences between default bridge and User-defined bridge networks

User-defined bridges provide automatic DNS resolution between containers

Containers on the default bridge network can only access each other by IP addresses, unless you manually create links using the --link option, which is considered legacy. These links need to be created in both directions, so this gets increasingly more complex the more containers there are to communicate.
On a user-defined bridge network, containers can resolve each other by name or alias. Imagine an application with a web front-end and a database back-end. If you call your containers web and db, the web container can connect to the db container using the hostname db.
User-defined bridges provide better isolation

All containers without a --network specified, are attached to the default bridge network, which can be a risk, as unrelated stacks/services/containers are then able to communicate.

Using a user-defined network provides a scoped network in which only containers attached to that network are able to communicate.
Containers can be attached and detached from user-defined networks on the fly

To remove a container from the default bridge network, you need to stop the container and recreate it with different network options.

But you can connect or disconnect containers from user-defined networks on the fly.
Each user-defined network creates a configurable bridge

If your containers use the default bridge network, you can configure it, but all the containers use the same settings, such as MTU and iptables rules. In addition, configuring the default bridge network happens outside of Docker itself, and requires a restart of the Docker Daemon.

User-defined bridge networks are created and configured using docker network create command. If different groups of applications have different network requirements, you can configure each user-defined bridge separately.
Linked containers on the default bridge network share environment variables

Originally, the only way to share environment variables between two containers was to link them using the --link flag. Although this is not possible with user-defined networks, there are better alternatives for sharing environment variables like:
```
- sharing files using Docker Volumes
- sharing variables from a `docker-compose` configuration when using Docker Compose
- using Docker Secrets and/or Docker Configs when using Docker Swarm Services
```

How to use the `bridge` driver?

I encourage you to follow along with the hands-on lab that follows.

Demo: Default bridge network with cli

Let’s start off by running two nginx containers (in the background with -d) named app1 and app2:

docker run -d --name app1 nginx:alpine
docker run -d --name app2 nginx:alpine

As discussed previously, when no particular --network is specified from the cli, the default bridge network is used.

We can check that nginx is in fact running on both containers by using docker exec to run curl localhost on both containers:

docker exec app1 curl localhost
docker exec app2 curl localhost

On success, this would print out the index.html page of nginx.

Footnotes:

docker exec is used to execute a command inside a running container.
nginx runs on port 80 inside its container.
curl localhost is the same as curl http://localhost:80

Now let’s try to communicate between the containers.

Trying to reach app2 from app1 using the hostname app2 will not work since DNS doesn’t work out of the box with the default bridge.

docker exec app1 curl app2

Let’s get the IP address of app2 with:

docker inspect -f '{{range.NetworkSettings.Networks}}{{.IPAddress}}{{end}}' app2

On my system, it is 172.17.0.3 (could be different in yours).

If we now try to reach app2 from app1 again, but with the IP address:

docker exec app1 curl 172.17.0.3

We should see that it works!

Using IP addresses is neither reliable nor easy to maintain since more containers could be added to the default network.

If we used the legacy --link option, we could have used the name of the container instead of the IP. But there are major drawbacks to using the default bridge (as discussed in the previous section), and using the --link is discouraged.

So, let’s directly go to…

Demo: User-defined bridge network with cli

A user-defined bridge network has to be created before it can be used. So let’s create one named my-bridge:

docker network create --driver bridge my-bridge

NOTE: since bridge is the default network driver, specifying --driver bridge in the command is optional.

We will run two containers again and name them app3 and app4:

docker run -d --name app3 --network my-bridge nginx:alpine
docker run -d --name app4 --network my-bridge nginx:alpine

Again, let’s test if nginx is running properly on both containers:

docker exec app3 curl localhost
docker exec app4 curl localhost

On successfully printing the index pages, let’s move on to communication between the containers.

If we try to reach app4 from app3 using the hostname app4:

docker exec app3 curl app4

We will see that it will succeed!

The same thing will also work when reaching app3 from app4:

docker exec app4 curl app3

Will we be able to reach app2 from app3 using DNS?

docker exec app3 curl app2

Nope.

What about using app2’s IP address? (connection timeout added to not keep curl waiting indefinitely)

docker exec app3 curl 172.17.0.3 --connect-timeout 5

No again.

We cannot reach app2 from app3 (or app4) and vice versa because they are on two separate networks, the default bridge and our user-defined my-bridge.

Demo: publishing container ports

Containers connected to the same bridge network effectively expose all ports to each other. For a container port to be accessible from the host machine or from hosts of external networks, that container port must be published using the -p or --publish flag.

Let’s see this in action.

All the existing containers have nginx running inside them separately on port 80 of each container, but none of them can be reached from the host’s port 80 yet. We can confirm by running curl on the host:

curl localhost

Assuming you didn’t have any applications running on port 80, curl should return an error like “Connection refused”.

If we want the container’s application (in this case nginx) to be reachable from the host, we can publish its port (i.e. map the container's port to a host port).

docker run -d --name app5 -p 3000:80 nginx:alpine

This will map port 3000 on the host to port 80 inside the container.

If we now try to curl on port 3000 of the host:

curl localhost:3000

We should see the nginx index page.

Nice work! :)

Clean up

We can remove all containers on the system using the following command:

docker rm -f $(docker ps -aq)

NOTE: this will also remove any containers you had in your system previously before this tutorial.

Let’s also remove the my-bridge as well.

docker network rm my-bridge

All clean! ✨

When to use the `bridge` driver? - possible use cases

When Multi-Container Networking is required on a single host

Networks created by using the bridge driver are contained within a single host, therefore it is ideal for most workloads that run on a single machine, like an engineer's local development environment or a company that runs multiple applications on a single server, etc.
When using Docker Compose

Docker Compose simplifies running multi-container workloads, especially on a single machine. It does so by using a YAML configuration file (docker-compose.yaml) like the following example:
```
version: "3.1"

services:
    nginx:
        image: nginx:alpine
        ports:
            - 80:80
    web:
        image: myapp-custom-image
    mysql:
        image: mysql
# ...
```
This file specifies a set of services that use docker images to run containers, and bridge networks are setup to enable inter-container communication. Compose creates and uses user-defined bridge networks out of the box when its services are first created using the docker-compose up command. It also removes the user-defined bridge networks created when services are taken down with docker-compose down.

NOTE: We will learn more about the awesome Docker Compose tool in an upcoming post, so, stay tuned.

If you know of any other use cases for bridge driver, please let us know.

Limitations of the `bridge` driver

Limited to a single host: The bridge driver provides seamless network isolation for containers on a single host only. If networking between multiple hosts is required, either the Overlay driver (recommended) or some custom OS level routing has to be implemented.
bridge driver is slower than host driver: Since an internal network has to be created and ports have to be mapped (via NAT/userland-proxy) for bridge driver, it incurs more overhead in terms of raw network performance at the cost of providing better isolation than the host driver. If you are not running multiple containers on a single production machine or if a very high network throughput is required, then the host driver might provide a better alternative to bridge due to its faster performance.
default bridge drawbacks: As discussed previously in the section ‘Differences between default bridge and User-defined bridge networks’, the default bridge is far inferior to user-defined ones. The default bridge network is considered a legacy detail of Docker and is not recommended for production use.

Conclusion

In this blog, we learnt about the bridge network driver in docker - what it is, how to use it, and some possible use cases and limitations.

By creating an internal network, the bridge driver simplifies network isolation on a single host. But its drawbacks are to be kept in mind, especially when using it in production.

I hope I could make things clearer for you, be it just a tiny bit.

In the next blog, we will learn about the overlay driver - which simplifies multi-host container networking.

Thanks for making it so far! 👏

See you at the next one.

Till then…

Be bold and keep learning.

But most importantly,

Tech care!

The Host Network Driver | Networking in Docker #5

Farhim Ferdous — Sun, 13 Mar 2022 01:26:37 +0000

Learn what the Host driver is, how it provides the best performance, how to use it, possible use cases and limitations

Link to video: https://www.youtube.com/watch?v=qice7-Cfgzw

What is the host network driver in Docker and how does it provide the best network performance?

This blog will try to answer that (and more) as simply as possible.

Introduction

This blog is the fifth one in a series for Docker Networking.

If you are looking to learn more about the basics of Docker, I’ll recommended checking out the Docker Made Easy series.

Here’s the agenda for this blog:

What is the host network driver?
How to use it?
When to use it? - possible use cases
its limitations

Here are some quick reminders:

A Docker host is the physical or virtual machine that runs the Docker daemon.
Docker Network drivers enable us to easily use multiple types of networks for containers and hide the complexity required for network implementations.

Alright, so...

What is the `host` driver?

When a container uses the host network driver, the container shares the network stack (namespace) of its host.

This means the network of the container is not virtualized, making the container appear as if it is the host itself, from a networking perspective.

A direct consequence of this is that - if a container using the host driver publishes a port, let’s say 8000, then that port will also be published on the host machine. If that port is already in use, then the container will not start successfully.

However, in all other ways - like storage, process, and user namespace, the container is isolated from the host.

NOTE: the host driver is only supported on Linux as of now i.e. host driver is not available on Mac or Windows platforms.

How to use the `host` driver?

Example without specifying any network - using default bridge driver

Let’s start off by running an nginx container named app1 in the background (-d) without specifying any network:

docker run -d --name app1 nginx:alpine

The nginx container by default maps onto port 80 inside the container. Docker uses the default bridge network driver if no driver is specified.

If we use curl localhost inside the container, we’ll see nginx the nginx index page successfully:

# works
docker exec app1 curl localhost

NOTE: Did you know port 80 is the default HTTP port? Therefore curl localhost automatically refers to curl http://localhost:80.

But if we try to curl localhost from the host machine, it will fail:

# fails
curl localhost

This is because we cannot reach the nginx container from the host, as the host and the container are using separate network namespaces (when using the default bridge network).

The solution is port mapping.

Let’s remove app1 and create it again using the port mapping -p option:

docker rm -f app1

docker run -d --name app1 -p 2000:80 nginx:alpine

The port mapping option (-p 2000:80) instructs Docker to map port 2000 on the host to port 80 on the container.

Let’s check out results using curl:

# fails
curl localhost

# works
curl localhost:2000

# works
docker exec app1 curl localhost

We’ll learn more about bridge drivers in the next part of this series.

Example using host driver

As we’d learnt, when using the host driver, the network stack is the same for both the container and host, so there’s no need for port mapping. Let’s check it out.

NOTE: you need a Linux machine to use the host driver.

We’ll run a similar nginx container named app2 using the host driver:

docker run -d --name app2 --network host nginx:alpine

Curling localhost will now work:

# works
curl localhost

Can you guess why?

The answer will be clear if we...

Check differences in IP configuration

Let’s first look at IP addresses assigned to all network interfaces on the host machine:

ip addr

Note the number of interfaces and the associated IPs.

What happens if we execute the same command inside the app1 container (the one using the default bridge driver)?

docker exec app1 ip addr

We should certainly see differences between the configuration in the host machine and app1, since they are using different network namespaces.

Now let’s try that for app2 container (the one using the host driver):

docker exec app2 ip addr

You’ll notice that the network configuration is exactly the same as for the host machine.

This is a clear indication that host driver makes the container share the host machines network.

One consequence of using the host driver is...

Port conflicts

You cannot run multiple containers that bind to the same port using host driver.

To demonstrate this, we’ll try to run a very similar nginx container to app2, but named as app3:

docker run -d --name app3 --network host nginx:alpine

If we check the logs:

docker logs -f app3

We’ll notice that nginx failed to start because the port it tried to bind to (80) was already in use.

We can confirm that app3 exited:

docker ps -a

Cleanup

To end the hands-on lab, we’ll clean up the containers:

docker rm -f app1 app2 app3

Now that we have seen it in action, let’s learn...

When to use the `host` driver? - possible use cases

The following are some use cases where the host driver could be suitable.

When the highest network performance is required

The host network driver provides the best network performance compared to the other drivers since it uses the network namespace of the host machine directly and does not require port-mapping / network address translation (NAT).
When a single container needs to handle a large number of ports

If our workload demands a container to handle a large number of ports, then it might be preferable to use the host network directly, instead of mapping each port to the host one by one.
When network isolation is not required

This can actually simplify our networking since we won’t need to bother with port mapping or NAT.

Given its properties in mind, we should be aware of the...

Limitations of the `host` driver

Lack of network isolation

As the container’s network is not virtualized using host, it may not provide the level of isolation required for many multi-container workloads.

Additionally, any networking configuration applied to the host machine will also be applied to the container. If the container is deployed in privileged mode, the container could reconfigure the host’s network stack as well.

This tight binding with the host machine could raise bugs or security issues.
Port conflicts

As we have seen, we cannot run multiple containers that use the same port when using the host driver.
Only works on Linux machines

The host driver is only supported on Linux, not on Docker Desktop for Mac or Windows.

Conclusion

In this blog, we learnt about the host network driver in docker - what it is, how to use it, some possible use cases and its limitations.

By not providing network isolation, the host driver provides the best performance and simplicity. But its drawbacks are to be kept in mind when using the host driver in production.

I hope I could make things clearer for you, be it just a tiny bit.

In the next blog, we will learn about the bridge driver - the default network driver, that is likely used the most in development environments.

Thanks for making it so far! 👏

See you at the next one.

Till then…

Be bold and keep learning.

But most importantly,

Tech care!

The None Network Driver | Networking in Docker #4

Farhim Ferdous — Sat, 12 Feb 2022 11:02:50 +0000

Learn what the None driver is, how it enables network isolation, how to use it, and possible use cases

Link to video: https://www.youtube.com/watch?v=LmZMTQd_y9M

What is the none network driver in Docker and how can it help me run secure, network-isolated containers?

This blog will try to answer that as simply as possible.

Introduction

This blog is the fourth one in a series for Docker Networking.

If you are looking to learn more about the basics of Docker, I’ll recommended checking out the Docker Made Easy series.

Here’s the agenda for this blog:

What is the none network driver?
How to use it?
When to use it? - possible use cases

Let’s start with a quick refresher: what are network drivers?

Network drivers enable us to easily use multiple types of networks for containers and hide the complexity required for network implementations.

So...

What is the `none` network driver?

The none driver simply disables networking for a container, making it isolated from other containers.

Within the container, only the loopback interface is created to enable inter-process-communication, by using the IP address 127.0.0.1 mapping to the name localhost.

But the absence of other network interfaces like eth0 means, neither can it reach external networks nor can external networks reach it.

This makes the none network driver very secure. But it also limits its usability, as we usually wish to enable communication between containers.

We’ll later look at some possible use cases for none.

For now, let’s learn...

How to use the `none` network driver?

First, we’ll run an interactive shell (sh) within an alpine Linux container named app1, without specifying a network:

docker run -it --name app1 alpine sh

NOTE: the -it flags connect your terminal’s input to the container’s one, giving you direct access to the shell running inside the container.

Docker uses the default bridge network driver if no driver is stated, which we will learn more about in an upcoming post.

But the important thing to note is that this container has access to the internet.

We can check that by pinging a web address like google.com:

ping google.com

or by running the apk update command:

apk update

They should work.

We can check the network interfaces and IP addresses by using:

ip addr

This shows detailed configurations for each network interface.

The two primary interfaces of interest are the loopback (having the familiar 127.0.0.1 address), and the ethernet interface (named eth0 or similar). The loopback allows processes within the container to communicate with each other, whereas, the ethernet interface within IP/inet address connects the container to a network.

NOTE: On some systems, additional tunnel interfaces (like tunl0) might show up; we can ignore them for now.

We can also check the presence of a route table that has a default route to a gateway:

route

All this was a way to confirm the presence of network connectivity. Don’t worry if you didn’t understand every single detail.

Okay.

Now let’s see how a container with the none driver behaves.

Let’s open up a new terminal tab/window and run a very similar container like before. This time we will specify the network to be none and name it app2.

docker run -it --name app2 --network none alpine sh

Trying to access the internet now shall fail - like pinging the google URL:

ping google.com

or apk update

apk update

If we try to ping localhost, it will surprisingly work:

ping localhost

This is because the loopback interface is present, as you can check with:

ip addr

But you will notice that there are no ethernet interfaces.

As well as no routes:

route

Thus, neither can app2 be reached from the outside nor vice versa.

The only way we can connect to app2 is by using the docker exec command, which is used to run a command inside a running container.

Open up a third terminal tab/window and run the following:

docker exec app2 ip addr

This will show the list of interfaces and IP addresses of app2 we had seen earlier.

If you want to start another interactive session, you can run another shell (sh) and use -it with exec:

docker exec -it app2 sh

To exit the shell, use ctrl+d or (control+d).

Once we are done playing around, we can clean up the containers using:

docker rm -f app1 app2

When to use the `none` network driver? - possible use cases

While I personally haven’t used the none driver in production much during the several years that I have been exposed to containers, it might still be useful for cases that require strict network isolation.

I could think of two niche use cases:

To run network-isolated applications that only perform file operations

This could be done by mounting volumes to the container, which then, in turn, could perform operations after certain intervals or by detecting file change events. Some examples include using a container for generating database backups, processing log files, etc.
To run a one-off command which requires network-isolation

Sometimes it might be required to just perform computations and print logs in a secure, network-isolated environment. Examples include running ci/cd jobs that export produced artifact files or testing suspicious scripts that could contain malware, etc.

According to the documentation, none may also be used in conjunction with a custom network driver.

Do you know of any other important use cases that I am missing? Please let me know as I’d love to add them here.

Conclusion

In this blog, we learnt about the none network driver in docker - what it is, how to use it, and some possible use cases.

By enabling network isolation, the none driver offers a secure environment for running processes. But it also limits its usability.

In the next blog, we will learn about the host driver - which provides the best network performance out of all 4 drivers! ⚡️

Thanks for making it so far! 👏

See you at the next one.

Till then…

Be bold and keep learning.

But most importantly,

Tech care!

Docker Network Drivers Overview | Networking in Docker #3

Farhim Ferdous — Mon, 10 Jan 2022 10:36:11 +0000

Learn how Docker Network Drivers allow easy Container network configuration by implementing the Container Network Model

Link to video: https://www.youtube.com/watch?v=vmgSIBzYoiE

What are Network Drivers in Docker? How do they enable easy network configuration for Containers?

This blog will try to answer that as simply as possible.

Introduction

This blog is the third one in a series for Docker Networking.

If you are looking to learn more about the basics of Docker, I’ll recommended checking out the Docker Made Easy series.

Here’s what we will discuss today:

The Container Network Model
Network Drivers: none, host, bridge & overlay

Okay.

Firstly, it’s good to know that networking in Docker is made possible by the...

Container Network Model

The Container Network Model (CNM) formalizes the steps required to provide networking for containers while providing an abstraction that can be used to support various types of networks.

CNM has 3 main components: Sandbox, Endpoint, and Network.

Sandbox: It contains the configuration of a container's network stack, which means management of the container's network interfaces, DNS settings, route tables, etc. A Sandbox may contain many endpoints from multiple Networks.
Endpoint: It joins a Sandbox to a Network. An Endpoint can belong to only one Network and it can belong to only one Sandbox if connected.
Network: It is formed by a group of Endpoints that are able to communicate with each other directly.

You can think of the Container Network Model as an abstract class that defines the required interface, whereas network drivers correspond to concrete classes implementing the interface.

This is, of course, an oversimplification so feel free to learn more about the design of CNM here.

As users of Docker, instead of the detailed implementations of CNM, we should learn more about how to properly use...

Network drivers

Network drivers are pluggable interfaces that provide the actual network implementations for Docker Containers.

Docker comes with several drivers out-of-the-box that provide core networking functionality for many use cases - like service discovery, encryption, multi-host networking, etc.

Then there are 3rd party drivers (by plugin providers) available for special use cases.

Lastly, one can even build their own custom drivers if available ones don't suffice (although that will rarely ever be required).

The 4 out-of-the-box network drivers are:

none
host
bridge
overlay

The driver can be specified by the --network option for the docker run command like this:

docker run -d --network host nginx

This command runs a nignx container using the host driver in the background (specified by the -d flag).

It’s interesting to note that a Container is generally unaware of the network driver it uses, except when using the none driver.

So, how do each of the drivers differ?

Reminder: a Docker host is a host/computer running the Docker daemon. You can learn more about Docker’s architecture here.

None Driver

The none driver simply disables networking for a container, making it isolated from other containers.

Host Driver

When using the host driver, the container shares the network stack of the Docker host - appearing as if the container is the host itself, from a networking perspective.

NOTE: the host driver is only supported on Linux as of now.

Bridge driver

The bridge driver creates an internal network within a single Docker host. containers placed within this network can communicate with each other but are isolated from containers, not on the internal network. bridge is the default driver when running single containers or when using docker-compose.

Overlay driver

The overlay driver creates a distributed network that can span multiple Docker hosts, and therefore is the preferred driver for managing container communication within a multi-host cluster. overlay is the default driver for Docker swarm services.

Conclusion

Alright.

Today we briefly learnt about the Container Network Model and the main Network Drivers that Docker provides.

In the upcoming blogs of this series, we will explore each of the Drivers in more detail.

Being pluggable makes drivers highly extensible & portable. As a result, they allow the use of various types of networks, and connect containers amongst themselves and with non-Docker workloads as needed.

I honestly find great joy when my fellow Engineers/Developers use Container technology effectively to build awesome things!

Thanks for making it so far! 🎉

See you at the next one.

Till then…

Be bold and keep learning.

But most importantly,

Tech care!

Basics of Networking | Networking in Docker #2

Farhim Ferdous — Wed, 08 Dec 2021 15:51:01 +0000

Understand the core IP network concepts useful for container networking

Link to video: https://www.youtube.com/watch?v=BnAozPAMXnc

What is an IP address or a network port? How do they relate to a network interface? 🤔

If you've ever struggled with networking concepts, this blog is for you.

Introduction

The purpose of this blog is to go over some basic networking topics, which will be specially useful to master networking for Docker containers.

If you're confident with your networking know-how, you can skip this blog and go directly to the next one.

But if you have no or very limited networking knowledge, or want to revise the core concepts, read on!

Here's what we will discuss:

Network Interfaces,
Internet Protocol (IP),
Public vs Private networks,
IP addressing,
Subnet mask and CIDR,
Network Ports,
DNS

NOTE: We will refer to computers/devices in a network as hosts.

To recap from part 1...

A network is formed when a group of devices is connected over a medium to share data.

But how do hosts connect to a network?

Network Interfaces

A network interface is the point of interconnection between a host and a network.

It can either be physical (like the one below) or virtual (emulated using software).

A local network (e.g. LAN) can be formed merely by connecting multiple hosts together (generally by using hardware devices like switches or hubs).

This eventually results in many local networks where communication between hosts is possible but only within that local network.

But how would one transmit data across networks?

Internet Protocol (IP)

IP (or Internet protocol) is a standard protocol for transmitting packets of data between hosts across networks, connected to form an IP network.

A host can send data to another one by specifying the destination host's IP address on the data packet. The packet of data can then be routed to the correct host via one or more routers.

A router is a special-purpose device with two or more network interfaces, connecting two or more networks, which allows it to forward data packets from one network to another.

Thus, IP allows us to form a network of networks i.e. internetwork.

Public vs private networks

A network is said to be public if anyone can access it from outside. The Internet is the best and probably the only pure example of such a network.

A private network is one to which access is restricted. Local networks like those in a company or school are examples of private networks.

Private networks offer security by not exposing hosts directly to the Internet.

IP addresses

Each host in an IP network can have one or more IP addresses, depending on how many network interfaces it has.

An IP Address is a numeric expression for identifying a particular host on a particular network.

10.2.0.10 is an example of an IP address.

32 bits are used to represent the address, so both addresses below are equivalent - the top one being decimal while the bottom one is binary.

   10   .   2    .   0    .  10
00001010.00000010.00000000.00001010

Since 8 bits are used to create each of the four-dot (.) separated segments, the value of each segment can range from 0 to 255.

Subnets masks & CIDR

Since an IP network can span multiple smaller networks (subnets), there needs to be a way to specify the network a host belongs to in order to route data packets to it.

The IP address of a host is often specified as 10.2.0.10/24, where /24 specifies the part of the address that belongs to the subnet i.e. the subnet mask. This is also called the CIDR notation.

For example, the IP 10.2.0.10/24 uses the first 24 bits to identify the network, whereas the rest of the 8 bits are used to identify hosts within that network. This allows 2^8 or 256 different IP addresses (practically fewer as some IPs are reserved).

The following shows the minimum and maximum IP address values for this subnet.

min: 00001010.00000010.00000000.00000000
max: 00001010.00000010.00000000.11111111

Therefore, the larger the subnet mask (i.e. the bigger the number after /), the fewer the hosts that subnet can accommodate.

Okay, we now know how data is transmitted to the correct network and host using IP.

But a host could be running many different applications. How can one host reach a particular application on a destination host?

Network Ports

A network port is a logical endpoint for specifying the correct host application/process being communicated with.

A port is identified for each Transport Layer protocol and IP address combination by a 16-bit unsigned number. The two most prominent Transport Layer protocols are User Datagram Protocol (UDP) and Transmission Control Protocol (TCP).

This means, for a host with a given IP address, there can be up to 65535 ports available for TCP and UDP each. Each application (or networked service) running on the host could then use zero or more of the ports that are available.

Port numbers from 0 to 1023 are called well-known or system ports and are reserved for standard networked services, like TCP port 80 for HTTP, TCP port 443 HTTPS, etc.

Just one more networking concept to remember...

Domain name system

The Domain name system (DNS) was invented to translate domain names to IP addresses (and vice versa), as it is difficult for us humans to remember IP addresses.

At the simplest level, DNS records are domain name to IP address mappings, like:

google.com    172.217.194.100

Records like these (and more complex ones) are distributed across multiple levels of name servers, which serve to provide this domain name to IP address translation.

Your host machine keeps a local cache of recently queried DNS records and asks name servers when it doesn't find the requested domain record locally.

You can check the IP addresses for google.com by typing this in your terminal:

nslookup google.com

Conclusion

In this blog, we briefly discussed - network interfaces, the Internet Protocol, public vs private networks, IP addresses, subnets, network ports, and DNS.

If we keep these concepts in mind, Docker container networking will be much easier to understand and apply.

Thanks for making it so far! 🎉

See you on the next one.

Till then…

Be bold and keep learning.

But most importantly,

Tech care!

Why is Networking important in Docker? | Networking in Docker #1

Farhim Ferdous — Tue, 02 Nov 2021 10:46:23 +0000

Introduce the need for learning about Docker Networking and an overview of this series

Link to video: https://www.youtube.com/watch?v=9VNI0r4k0jo

Docker enables massive innovation in today's fast-paced IT industry.

And a lot of this is made possible by connecting multiple containers together. 🔌

If you, as an Engineer or Developer, ignore the networking part of Docker, you will miss out on arguably one of its most powerful capabilities.

But, before we understand "WHY" it's important, let's first answer...

What is a network?

A network is formed when a group of devices is connected over a medium to share data.

The devices (or computer systems) would be connected to the network by some form of physical or virtual interface.

The medium could be wired or wireless, be it cables, telephone lines or radio waves, etc.

But the main purpose remains to connect multiple systems together for some form of data sharing.

So...

Why is networking important in Docker?

A Container, being a form of virtualization, acts as an isolated system from the Docker host and from other containers.

Oftentimes, we need to connect multiple containers to produce a useful outcome.

This is the answer to our "WHY".

Let's look at an example to make this clear...

Suppose we have a web application that requires a database. So we create two containers named web and db respectively.

The web application could be for example Django and its database could be MySQL, or whatever framework/library and database you prefer, that's not our concern.

The important thing is that, web needs to connect to db for storing, modifying, and retrieving data.

The database, in turn, would preferably use Docker Volumes to persist that data, since the data in a container is not persistent by default (learn more about Volumes and persistence in Docker here).

But, how would we achieve a connection between the two containers? 🤔

If both containers were connected to an IP network, they would each have a unique IP address on that network.

Let's say web is at IP 10.0.0.2, and db is at IP 10.0.0.3.

web could then connect to db using the IP 10.0.0.3 and we would live happily ever after! 🏝

Cool.

But how do we actually create this network in the first place?

Also, if we wanted to reach our web application from the host or from the Internet, how'd we do that?

The problems seem simple enough but implementing a solution doesn't sound like so.

Welcome to the complexity so common in networking. 🕸

Wait! ✋

Before you run away from this complexity, there's good news. 🏃‍♂️💨

There is no complexity for us to manage because Docker handles the network complexity for us! In fact, that is one of the primary advantages of using Docker.

Yay!!! 🙌

As users of Docker, we just need to know about the main network drivers, which are high-level interfaces that abstract away the network implementations.

Don't worry if that doesn't make sense yet, we'll be learning more about them in the upcoming sections of this series.

Here's the...

Structure of this series - Networking in Docker

(current) Why is Networking important in Docker?
The Basics of Networking
Container Network Model & Drivers Overview
Network driver: none
Network driver: host
Network driver: bridge
Network driver: overlay
Summary, Differences & What's next?

Conclusion

In this short introduction, we briefly discussed what a network is and why it's important in Docker.

We also looked at the structure of this series - Networking in Docker.

Knowing about the basics of networking can definitely help, not just with Docker but as an Engineer/Developer in general because so much of our work is made possible by the Internet, which is a network itself!

So, that's what we will learn in the next part - The Basics of Networking.

I'm super excited about this series because I myself am also learning a lot about networking from it! See you on the next one. 👋

Till then...

Be bold and keep learning.

But most importantly,

Tech care!

DEV Community: Farhim Ferdous

What is a Server?

Introduction

What is a Server?

Why should we learn about Servers?

Why do Servers exist?

Server Computers VS Desktop Computers

How do Servers Work?

Types of Servers

Conclusion

What is DevOps?

Introduction

What is DevOps?

DevOps Practices

1) Break down organizational silos to empower cross-functional teams

2) Release short and incremental changes

3) Adopt microservice architectures

4) Version control everything

5) Automate everything

DevOps Technologies

Why learn DevOps?

1) Massive learning opportunity

2) High-impact role

3) High salary & exploding demand

Do I HAVE to be a ‘DevOps Engineer’?

The Challenges in learning DevOps

How TechMormo helps you master DevOps?

Conclusion

ভার্চুয়ালাইজেশন কি? (Virtualization) | ডকার কন্টেইনার vs ভার্চুয়াল মেশিন

উদাহরণ - কি সমস্যার সমাধান দরকার?

ভার্চুয়ালাইজেশন এর সংজ্ঞা (Definition)

ভার্চুয়ালাইজেশন এর উপকারিতা (Benefits)

ভার্চুয়ালাইজেশনের প্রকারভেদ (Types)

তুলনা: কন্টেইনার, ভার্চুয়াল মেশিন ও বেয়ার মেটাল

বেয়ার মেটাল (Bare metal)

ভার্চুয়াল মেশিন (VM)

কন্টেইনার (Container)

উপসংহার (Conclusion)

What is Docker Compose? (with demo)

Introduction

What is Docker Compose?

Why Docker Compose exists? How does it differ from docker run?

How to use Docker Compose?

Step 1: Install Docker & Docker Compose

Step 2: Create a sample web application

Step 3: Run and test the application

Step 4: Modify the application

Step 5: Clean up

When to use Docker Compose? - use cases

Conclusion

Docker Networking Summary | Networking in Docker #8

Introduction

Summary

Conclusion & what’s next?

The Overlay Network Driver | Networking in Docker #7

Introduction

What is the overlay driver?

How to use the overlay driver?

Initialize the swarm

The default overlay network - ingress and routing mesh

User-defined Overlay Networks

Clean up

When to use the overlay driver? - possible use cases

Limitations of the overlay driver

Conclusion

The Bridge Network Driver | Networking in Docker #6

Introduction

What is the bridge driver?

Differences between default bridge and User-defined bridge networks

How to use the bridge driver?

When to use the bridge driver? - possible use cases

Limitations of the bridge driver

Conclusion

The Host Network Driver | Networking in Docker #5

Introduction

What is the host driver?

How to use the host driver?

When to use the host driver? - possible use cases

Limitations of the host driver

Conclusion

Why Docker Compose exists? How does it differ from `docker run`?

What is the `overlay` driver?

How to use the `overlay` driver?

The default overlay network - `ingress` and routing mesh

When to use the `overlay` driver? - possible use cases

Limitations of the `overlay` driver

What is the `bridge` driver?

How to use the `bridge` driver?

When to use the `bridge` driver? - possible use cases

Limitations of the `bridge` driver

What is the `host` driver?

How to use the `host` driver?

When to use the `host` driver? - possible use cases

Limitations of the `host` driver

What is the `none` network driver?

How to use the `none` network driver?

When to use the `none` network driver? - possible use cases