DEV Community: Oni Toluwalope

HOW A 30-MINUTE TASK TURNED INTO A 9-HOUR TROUBLESHOOTING NIGHTMARE

Oni Toluwalope — Tue, 12 Aug 2025 17:17:33 +0000

As developers, and engineers, we all have those "quick tasks" we expect to knock out in half an hour, tops. A few weeks back, my goal was simple: set up SonarQube locally to get a better understanding of its configuration. While I'd used it before on some IBM cloud machines, I wanted hands-on experience with a local setup.

I'd made a few minor code changes and wanted to see how SonarQube would flag them—a process I anticipated would take around 30 minutes from start to finish. Little did I know, this seemingly straightforward task would plunge me into a rabbit hole of configuration errors and resource limitations, stretching my troubleshooting skills to their limit over a grueling nine hours.

The first sign of trouble appeared quickly. SonarQube simply refused to start. When I tried to access it through my browser, all I got was a connection refused error. My initial reaction was slight annoyance, but nothing alarming. "Probably just a small hiccup," I thought, diving into what I assumed would be a quick configuration tweak.

My First Mistake: I Didn't Check the Logs Immediately

Instead of checking the logs, I started fiddling with network settings, assuming the issue was a port conflict or a firewall rule. I spent a good hour going down this path, tweaking configurations that ultimately had nothing to do with the actual problem. Had I taken a moment to peek at the SonarQube logs right away, I would have seen a clear error message screaming about insufficient memory map areas.

Eventually, after my fruitless network explorations, I investigated the console output more closely. That's where I was greeted by a cryptic error message mentioning vm.max_map_count. This was a parameter I'd vaguely encountered before, but its exact purpose eluded me.

My Second Mistake: I Misinterpreted What `vm.max_map_count` Meant

This leads to my second blunder: I initially thought vm.max_map_count was directly related to the total memory available to my Kali Linux Virtual Machine. I misinterpreted the "vm" as "Virtual Machine" rather than "virtual memory." Based on this flawed understanding, I started adjusting the RAM size of my virtual machine in Oracle VirtualBox, thinking I needed to allocate more memory. This, of course, didn't solve the issue, and I spent another couple of hours chasing this red herring 😭😭😭😭😭.

It wasn't until I decided to research the error message more thoroughly that the moment of discovery came. I learned that this kernel parameter defines the maximum number of memory map areas a process can have. SonarQube, specifically its Elasticsearch component, requires a large number of these maps, and the default setting on my system was simply too low.

Even with this newfound understanding, I still hesitated. Modifying kernel parameters felt like venturing into the operating system's core, and I wanted to be sure I was making the right change. So, I spent some more time double-checking the documentation and searching for similar issues online.

My Third Mistake: I Didn't Increase `vm.max_map_count` Sooner

This was my third mistake: not increasing vm.max_map_count in a timely manner. While due diligence is important, my hesitation prolonged the outage unnecessarily.

Finally, I bit the bullet and decided to increase the vm.max_map_count value. I used the command:

sudo sysctl -w vm.max_map_count=524288

(You can also edit the sysctl.conf file, add the line vm.max_map_count=524288, and apply the changes using sudo sysctl -p.)

I felt a surge of hope as I restarted SonarQube.

My Fourth Mistake: I Overlooked Storage and Under-Allocated Resources

My elation was short-lived. SonarQube started, but it was incredibly slow and unstable, frequently crashing with out-of-memory errors. I wondered what was wrong since I saw that 512 MB of memory was the minimum required for SonarQube to run, and I had set it to that value. Out of frustration, I ended up increasing the vm.max_map_count to 2,097,152 KB (2 GB), of which the initial 524,288 was sufficient, but I was still faced with the same problems and, once again, did not read the logs on time.

I tried different things I saw online regarding the out-of-memory error, and nothing worked. Eventually, checking the logs led me to another realization: I didn't have enough storage on my Kali Linux virtual machine. The virtual disk was almost full, leaving insufficient space for SonarQube to operate efficiently, especially with Elasticsearch indexing and storing data.

This was the final piece of the puzzle. I had to shut down the virtual machine, resize the virtual disk in my virtualization software, delete some unnecessary files, and then expand the filesystem within Kali Linux.

After this final step, I restarted the entire system and, with bated breath, started SonarQube again. This time, it worked! The web interface loaded quickly, the code analysis ran smoothly, and my initial 30-minute task was finally completed—nine hours later, a significant portion of which was also spent waiting for my slow, and patient testing computer to respond.

Key Takeaways from the Marathon Troubleshooting Session

This experience, though frustrating, was a valuable reminder of several key lessons:

Check the Logs First: They are your best friends. Don't overlook them in your initial attempts to diagnose a problem.
Understand Error Messages: Don't make assumptions. Research cryptic error messages to grasp the underlying issue accurately.
Address System Requirements: Ensure your system meets the minimum and recommended requirements for the software you are running, including memory, storage, and kernel parameters. Don't assume default settings are sufficient.
Be Decisive (But Informed): While research is crucial, don't hesitate to implement a likely solution once you have a good understanding of the problem.
Consider the Environment: If you're working in a virtualized environment, remember to check the resources allocated to the virtual machine itself.

What was supposed to be a quick 30-minute code quality check turned into a nine-hour deep dive into system configuration and resource management. While I wouldn't wish this marathon troubleshooting session on anyone, the satisfaction of finally resolving the issue, coupled with the valuable lessons learned—like the pain of working under tight compute resources—made it (almost) worth the time.

Now, if you'll excuse me, I need a hot bowl of chicken pepper soup!

Configuration Management: 12 factor, Factor III

Oni Toluwalope — Thu, 03 Jul 2025 07:18:03 +0000

As a developer or DevOps engineer, you're well aware of the importance of keeping your application secure and scalable. One crucial aspect of achieving this is by following the 12-Factor App methodology, specifically Factor III: Configuration. In this article, we'll dive into the world of configuration management, exploring its benefits, best practices, and the importance of prioritizing it in your development workflow.

Why Configuration Management Matters

Configuration management is a critical component of application development, and its importance cannot be overstated. Here are three key reasons why configuration management matters:

Security: By storing configuration in the environment, separate from code, you can prevent sensitive credentials like API keys and database passwords from being exposed in version control. This is especially important in today's world, where security breaches can have devastating consequences.
Portability: With configuration management, you can deploy the same build artifact across different environments (dev, staging, production) by changing environment variables. This eliminates the need for code changes, making it easier to manage your application's lifecycle.
Flexibility: Configuration management allows you to easily manage settings for different deployment stages or scaling scenarios. This flexibility is essential in today's fast-paced development environment, where applications need to be adaptable and responsive to changing requirements.

Best Practices

So, how can you implement effective configuration management in your application? Here are some best practices to get you started:

Use .env files for local development: Keep sensitive data separate from your codebase by using .env files for local development. This will help prevent accidental exposure of sensitive credentials.
Consider using a secrets management tool: Tools like HashiCorp Vault, AWS Secrets Manager, or Azure Key Vault can help you manage sensitive data and configurations securely.

The Case for Alternative Secrets Management Tools

While HashiCorp Vault Secrets is a powerful tool for secrets management, it's worth noting that HashiCorp will be discontinuing HCP Vault Secrets. After June 30, 2025, new customers won't be able to purchase it, and existing customers will have until August 27, 2025. This means it's essential to explore alternative solutions that can provide similar functionality.

Some popular alternatives to HashiCorp Vault include:

AWS Secrets Manager: A fully managed service that securely stores and manages sensitive data in AWS environments.
Azure Key Vault: A cloud-based solution for safeguarding cryptographic keys and other secrets.
Doppler: A multi-cloud SecretOps Platform for enterprise-scale secrets management.
Akeyless Vault: A cloud-native, SaaS-first secrets management solution that eliminates traditional vault-based architectures.

Make Configuration Management a Priority

Don't risk exposing sensitive credentials in your code. By prioritizing configuration management, you can ensure your application's security and scalability. Learn from common mistakes and adopt best practices to safeguard your secrets.

In conclusion, configuration management is a critical aspect of application development that requires careful consideration. By following best practices and leveraging the right tools, you can ensure the security and scalability of your application. Make configuration management a priority today and take the first step towards building more secure and scalable applications.

Key Points

Prioritize configuration management to ensure application security and scalability.
Use .env files for local development and consider secrets management tools for production environments.
Explore alternative solutions to HashiCorp Vault, such as AWS Secrets Manager, Azure Key Vault, Doppler, and Akeyless Vault.

By following these best practices and leveraging the right tools, you can ensure the security and scalability of your application.

DevOps #12Factor #ConfigurationManagement #AppSec #SecurityBestPractices #EnvironmentVariables

Dependency Management 101: Mastering the 12-Factor App's Factor II

Oni Toluwalope — Wed, 14 May 2025 21:28:08 +0000

Tackling the Pain Points of Deployment Failures

Who hasn't experienced the frustration of a deployment failure due to a missing library or the wrong version on the server? Factor II of the 12-Factor App addresses this issue head-on, and it's a game-changer for developers. In this article, we'll dive into the world of dependency management and explore the two crucial aspects that can make or break your application's reliability.

The Power of Explicit Declaration

Explicit declaration is the practice of listing every single dependency in a manifest file. This approach ensures that:

Dependencies are clearly defined: No more wondering what's required for your application to run.
Versions are specified: Avoid version conflicts and ensure reproducibility.
Dependencies are easily manageable: Update, add, or remove dependencies with confidence.

Some popular manifest files include:

requirements.txt for Python
go.mod for Go
package.json for Node.js

A Relatable Analogy

Imagine you're preparing a meal (application) that requires specific ingredients (dependencies) like flour, sugar, and eggs. You write down all the ingredients you need and their quantities (versions) on a shopping list. Similarly, explicit declaration in dependency management involves listing all dependencies and their versions in a manifest file.

The Importance of Isolation

Isolation is crucial for maintaining a consistent and reliable environment. Tools like:

Python's virtualenv
Go modules
Docker containers

help keep dependencies separate from:

System dependencies: Avoid polluting the system with project-specific dependencies.
Other projects: Prevent dependency conflicts between projects.

Building on the Analogy

Just as you separate your items into different bags for each meal or occasion, isolation in dependency management achieves a similar goal. It keeps project-specific dependencies separate from system dependencies and other projects, ensuring that your application runs smoothly and consistently.

The Benefits of Dependency Management

The combination of explicit declaration and isolation dramatically reduces environment inconsistencies and makes deployments smoother. Although it may seem minor, neglecting dependency management can lead to significant time waste, and I've learned firsthand that it's essential for reliable software delivery.

Best Practices for Dependency Management

To take your dependency management to the next level:

Use a dependency manager: Choose a tool that fits your project's needs, such as npm, yarn, or gradle.
Keep dependencies up-to-date: Regularly update dependencies to ensure security and stability.
Test dependencies: Verify that dependencies work as expected in your application.

What Tools Do You Rely On?

Share your favorite dependency management tools and techniques in the comments below. Let's learn from each other and improve our dependency management skills.

By mastering dependency management, you'll be well on your way to building reliable, scalable, and maintainable applications. Happy coding!

Decoding the Shebang (#!) in Shell Scripts: A Deep Dive

Oni Toluwalope — Wed, 14 May 2025 21:20:39 +0000

When working with shell scripts, you've likely encountered the mysterious #! line at the top. But what does it do? In this post, we'll explore the Shebang, its purpose, and best practices for using it in your shell scripts.

What is the Shebang?

The Shebang, also known as Hashbang or "sha-bang", is a special sequence of characters (#!) that specifies the interpreter to use when running a script directly. It's typically followed by the path to an interpreter, such as /bin/bash or /usr/bin/env bash.

How Does the Shebang Work?

When you run a script directly (e.g., ./my_script.sh), the operating system's program loader reads the Shebang line to determine which interpreter to use. The interpreter is then responsible for executing the commands within the script.

Types of Shebang

#!/bin/bash: Specifies the Bash shell interpreter located at /bin/bash.
#!/usr/bin/env bash: Searches for the Bash executable in the user's PATH, making the script more portable across systems.

Why Use #!/usr/bin/env bash?

Using #!/usr/bin/env bash provides better portability across systems where Bash might reside in a different location (like /usr/local/bin/bash). This approach ensures your script works regardless of the Bash installation location.

Consequences of Forgetting the Shebang

Without the Shebang, you'd need to specify the interpreter when running the script, like bash sample_script.sh. This can be inconvenient, especially when working with complex scripts or automating tasks.

Best Practices

Use #!/usr/bin/env bash for better portability.
Make your scripts executable with chmod +x my_script.sh.
Run your scripts directly with ./my_script.sh.

By understanding the Shebang and using it effectively, you can write more efficient, portable, and convenient shell scripts.

Conclusion

The Shebang is a critical component of shell scripting, allowing you to specify the interpreter and make your scripts directly executable. By following best practices and using #!/usr/bin/env bash, you can ensure your scripts work across different systems and environments.

ShellScripting #Bash #Linux #Unix #Programming #Automation #DevOps

Decoding the Shebang (#!) in Shell Scripts: A Deep Dive

Oni Toluwalope — Thu, 08 May 2025 22:15:02 +0000

What is the Shebang?

How Does the Shebang Work?

Types of Shebang

#!/bin/bash: Specifies the Bash shell interpreter located at /bin/bash.
#!/usr/bin/env bash: Searches for the Bash executable in the user's PATH, making the script more portable across systems.

Why Use #!/usr/bin/env bash?

Consequences of Forgetting the Shebang

Best Practices

Use #!/usr/bin/env bash for better portability.
Make your scripts executable with chmod +x my_script.sh.
Run your scripts directly with ./my_script.sh.

#!/usr/bin/env bash

echo "Hello, World!"

By understanding the Shebang and using it effectively, you can write more efficient, portable, and convenient shell scripts.

Conclusion

ShellScripting #Bash #Linux #Unix #Programming #Automation #DevOps

Foundation of Consistency: The 12-Factor App's 'Codebase' Principle

Oni Toluwalope — Sun, 04 May 2025 22:18:19 +0000

The 12-Factor App methodology provides a robust framework for building scalable and maintainable applications. The first factor, "Codebase," sets the foundation for a well-structured development process. In this post, we'll dive into the importance of maintaining one codebase per application, tracked in a version control system like Git.

Let's define what a codebase is

A codebase, also known as a code repository or source code repository, is a collection of source code files and related assets that make up a software application or system. At it's core, it is the entire body of code that defines the functionality, behavior, and structure of a software project.

Why One Codebase Matters

Having a single codebase per application ensures:

Traceability: Every deploy originates from a specific, known commit hash, making it easier to track changes and debug issues.
Consistency: The same code forms the basis for all deployments, reducing environment-specific variations and ensuring that your application behaves consistently across development, testing, and production environments.
Collaboration: A single codebase provides a single source of truth for all developers, promoting collaboration and reducing confusion.

The Benefits of a Single Codebase

While you might have multiple instances or deploys of your application, they should all stem from this single, version-controlled codebase. This approach ensures that:

All changes are tracked and auditable
Deployments are consistent and predictable
Developers can collaborate effectively

Best Practices

To implement this principle, follow these best practices:

Use a version control system like Git to track changes to your codebase
Maintain a single codebase per application
Ensure all deployments originate from the same codebase

By following the "Codebase" principle, you'll lay the foundation for a scalable, consistent, maintainable, and collaborative development process.

#12factorapp #Codebase #VersionControl #Collaboration #Scalability

Mastering Git: Understanding `git init` and `git clone` for Effective Version Control

Oni Toluwalope — Sun, 04 May 2025 21:59:34 +0000

As a developer, you're likely no stranger to Git, the powerful version control system that helps you manage changes to your codebase. However, for those just starting out or looking to solidify their understanding, two fundamental commands can often cause confusion: git init and git clone. In this post, we'll dive into the differences between these commands, when to use each, and how they fit into your overall Git workflow.

git init: Starting a New Project from Scratch

When you're beginning a new project and want to put it under version control, git init is the command you'll use. This command creates a new Git repository in your project directory, initializing it with a hidden .git folder that tracks changes to your code.

Here's a step-by-step breakdown of what happens when you run git init:

A new .git directory is created in your project folder, containing the necessary metadata for Git to track changes.
Your project is now under version control, and you can start committing changes.
No connection to a remote repository is made automatically. You'll need to add this manually using git remote add origin <URL>.

git clone <repository URL>: Getting a Local Copy of an Existing Project

On the other hand, when you want to obtain a local copy of a project that already exists on a remote server (like GitHub, GitLab, or Bitbucket), git clone is the way to go. This command downloads the entire project history and sets up a connection to the remote repository, allowing you to start working on the project locally.

Here's what happens when you run git clone <repository URL>:

The entire project history is downloaded to your local machine, including all commits, branches, and tags.
A connection to the remote repository is established, allowing you to push and pull changes.
The remote repository is automatically set up as the origin remote, making it easy to collaborate with others.

When to Use Each Command

So, when should you use git init versus git clone? Here are some scenarios to help you decide:

Use git init when:
- You're starting a new project from scratch and want to put it under version control.
- You want to create a new repository without connecting to a remote server.
Use git clone when:
- You want to obtain a local copy of an existing project from a remote server.
- You're collaborating with others on a project and need to get the latest changes.

Best Practices for Working with Git

Now that you understand the difference between git init and git clone, here are some best practices to keep in mind when working with Git:

Always initialize your project with git init or git clone to track changes and collaborate with others.
Use meaningful commit messages to describe the changes you've made.
Regularly push your changes to a remote repository to ensure your work is backed up and accessible to others.

By mastering git init and git clone, you'll be well on your way to becoming a Git pro and managing your projects effectively. Happy coding!

Additional Resources

Git documentation: https://git-scm.com/docs/git-init, https://git-scm.com/docs/git-clone
GitHub guides: https://guides.github.com/introduction/git-handbook/

Brief Intro to Operating Systems for Devops

Oni Toluwalope — Tue, 31 Dec 2024 13:13:50 +0000

An operating system provides a base for application software and act as an intermediary between the hardware, applications and the human user.

Some examples of operating systems used in modern day(2024): Android - developed by, iOS, MacOS, Windows, and Linux.

To understand what an Operating System does we need to understand the basic structure of a computer.

From bottom up, the computer consist of:

Hardware: Resources like CPU, Memory: RAM and ROM, Motherboard, Input-Output devices: Mouse, display, keyboard.

Application Softwares: They are softwares that are used to perfom specific tasks eg. Web browser, Word processors, Games

Imagine a world without OS where the user is to use the application software to interact with the computer hardware,
the user will have to tell the computer what to do in a step by step fashion using code.
For example the user wants to use a web browser called firefox,
the user will tell the hardware to load firefox to memory,
after loading it,
tell it to prepare the application to receive input,
display each character that is being typed on the monitor,
after displaying all characters, after displaying all characters,
you tell it you are done typing,
then you tell it again to convert it to packets,
after the data is being split into packet,

the user will request that the hardware open up the network ports,
then tell it to allow data outflow . . .

So, without the OS you have to manually dictate every step of what to do in form of code, and it will be very difficult to work with the computer if there is no operating system. And so this is the ease operating system brings.

If an Operating system was available all the user will do is to click on the program then type the address on the address bar then click enter, and the operating system takes the job of telling the hardware the step by step process on what to do. This makes the computer easier to use

So now it is safe to say an operating system manages computer hardware, provides a base for application software and act as an intermediary between the hardware, applications and the human user.

Main Functions of an operating system

Process management/ Task scheduling:
Memory Management: Handles running applications (processes), multitasking, and scheduling.
File System Management: Organizes and controls data storage and access
Input/Output management: Input/Output connection and flow

Other functions

Security management
Network management
Interfaces between hardware and software

Types of Operating Systems

Batch OS
Real Time OS (RTOS)
Time Sharing OS
Distributed OS
Network OS

Goals of an operating system

Ease of use
Allocation of resources
Efficiency

Why Operating Systems Matter in DevOps

Hosting Applications
Every application needs an OS to run. DevOps professionals often work with servers running Linux or Windows to host web applications, databases, and other services.
Linux is preferred for its stability, open-source nature, and extensive toolset. Windows Server is also common in enterprise environments requiring compatibility with Microsoft tools.
Containerization and Virtualization
Containers (e.g., Docker) and virtual machines (e.g., VMware, VirtualBox) rely heavily on OS-level features for isolation and resource management. Tools like Kubernetes orchestrate these containers, making OS knowledge indispensable.
Automation and Scripting
Automation in DevOps involves writing scripts and using configuration management tools. These often interact directly with the OS to:
Manage system users and permissions
Install software packages
Configure network settings
Monitoring and Troubleshooting
Understanding the OS helps in monitoring performance and diagnosing issues using tools like:
Linux: top, htop, dstat, journalctl
Windows: Task Manager, Event Viewer, PowerShell

Some Key Operating System Concepts for DevOps

Processes and Services
Processes: Running applications or tasks. Use tools like ps (Linux) or Task Manager (Windows) to monitor them.
Services: Background processes that perform specific functions, e.g., web servers (Nginx, Apache), loggers.
File Systems
Linux: Follows a hierarchical structure with directories like /home, /var, /etc.
Windows: Follows a flat directory structure that uses drive letters (e.g., C:) and organizes files accordingly.

Learn commands like ls, cd, cp, rm (Linux) or PowerShell equivalents for file management.

Networking
Operating systems manage network interfaces and routing. Key tasks include:
Configuring IP addresses
Managing firewalls (e.g., iptables, Windows Defender Firewall)
Troubleshooting with tools like ping, netstat, traceroute
User Management and Permissions
Linux: Uses file permission systems (chmod, chown) and role-based access.
Windows: Manages users through Computer Management, Control Panel or PowerShell.
Secure environments by limiting user access and enforcing best practices.
Package Management
Linux: Tools like apt (Ubuntu/Debian), yum (CentOS), handle software installation and updates.

Windows: Use the Windows Package Manager (winget) or Chocolatey.

Practical Applications of OS Knowledge in DevOps

Setting Up a Server
Example: Deploying a web application on Linux.
Install necessary packages (e.g., nginx, python3)
Configure the web server to serve the application.
Secure the server using firewalls and SSH keys.
Containerizing Applications
Example: Using Docker.
Understand base images (e.g., ubuntu:24.04 or alpine:latest).
Write Dockerfiles that leverage OS commands for setup.
Automating Tasks
Example: Scheduling backups with Cron (Linux) or Task Scheduler (Windows).
Write scripts to automate repetitive tasks.
Schedule them to run at specific intervals.
Monitoring and Logging
Example: Using system logs for troubleshooting.
Access logs in /var/log (Linux) or Event Viewer (Windows).
Use monitoring tools like Prometheus and Grafana for system metrics.

How to Start Learning Operating Systems for DevOps

Choose a specific OS (Linux Distribution is preferred)
Start with beginner-friendly distros like Ubuntu or CentOS.
Practice basic commands and explore the directory structure.
Set Up a Virtual Lab
Use VirtualBox or VMware to create virtual machines.
Experiment with Linux and Windows Server environments.
Learn Command-Line Basics
Master essential commands for file management, process monitoring, and networking.
Write Simple Scripts
Automate tasks using Bash (Linux) or PowerShell (Windows).
Gradually incorporate tools like Git, Ansible or Terraform.
Experiment with Containers
Install Docker and create basic containers.
Understand the relationship between the host OS and containerized applications.

OS knowledge empowers DevOps practitioners to work more effectively. By mastering basic OS concepts and gradually exploring advanced topics, you can build a solid foundation for your DevOps journey.

Docker vs Virtual Machines: What’s the Difference?

Oni Toluwalope — Mon, 30 Dec 2024 19:39:40 +0000

In the world of modern IT infrastructure, Docker and Virtual Machines (VMs) are popular technologies for application deployment and resource management. While they may seem similar at first glance, they differ significantly in architecture, use cases, and resource efficiency. This article explores the key differences between Docker and Virtual Machines, their advantages, and when to use each.

Let's begin from the components of an operating system, An operating system (OS) is the core software that manages a computer's hardware and software resources. There are two main components, the user space and the kernel.

User Space:
The user space is where user applications and utilities run, interacting with the kernel via system calls. It could be via a GUI or a command line interface.

Kernel: The kernel is core of the OS. It manages system resources like CPU scheduling, memory, file system, input output devices

What is a Virtual Machine?
A Virtual Machine (VM) is a virtualized instance of a complete operating system (OS) that runs on top of a hypervisor. VMs simulate the functionality of physical hardware, allowing multiple OS instances to run on a single physical machine. These OSes have their own User Space and Kernel, and that makes them function like a full OS.

A hypervisor allows multiple operating systems to run simultaneously on a single physical server.
It is like a layer of software that sits between the physical hardware (CPU, RAM, storage) and the guest operating systems. It manages and allocates resources to each guest OS, making it appear to each guest as if it has exclusive access to the hardware.

Key Features of Virtual Machines:

Full OS Virtualization: Each VM includes a full OS, its own kernel, and necessary libraries and binaries.

Resource Isolation: VMs provide strong isolation, as each runs in its own environment with dedicated virtual resources.

Hardware Independence: VMs abstract the underlying hardware, making them highly versatile for running different OS types.

What is Docker?

Docker is an example of container technology. It is a platform for developing, shipping, and running applications in lightweight, portable containers. Containers package an application and its dependencies into a single unit that can run reliably across different computing environments. In the case of Docker, Docker only has its own user space where applications run, but it shares the kernel with the underlying OS

Key Features of Docker:

Lightweight: Docker containers share the host operating system’s kernel, which makes them smaller and faster to start compared to VMs.

Portability: A Docker container can run on any system with Docker installed, regardless of the underlying hardware or OS.

Isolation: Containers operate in isolated environments, ensuring that applications do not interfere with each other.

Efficiency: By sharing the host OS kernel, Docker containers use fewer resources compared to VMs.

Architectural Differences

Docker Architecture:

Docker containers share the host OS kernel.

They include only the application and its dependencies.

Containers run as isolated processes on the host system.

VM Architecture: Each VM includes its own OS, kernel, and application stack.

VMs are managed by a hypervisor that abstracts the hardware.

Diagram:

Performance Comparison

Resource Utilization

Docker: Containers share the host OS kernel, resulting in lower overhead and better resource utilization.
VMs: Each VM requires its own OS, which increases memory and storage usage.

Startup Time

Docker: Containers start in seconds since they don’t require booting a full OS.
VMs: VMs can take minutes to boot because a complete OS startup is required.

Portability

Docker: Containers are highly portable across environments as long as Docker is installed.
VMs: Portability is limited due to dependencies on hypervisors and VM-specific formats.

Security

Docker:
Containers share the host OS kernel, which can pose security risks if the host is compromised.
Security relies on namespaces, control groups (cgroups), and additional tools like SELinux or AppArmor.
VMs:
VMs provide strong isolation as each has its own kernel.
Even if a VM is compromised, it is unlikely to affect others or the host system.

Use Case Scenarios
When to Use Docker:
Microservices Architecture: Ideal for deploying microservices as containers are lightweight and scalable.
CI/CD Pipelines: Frequently used in DevOps workflows for rapid testing and deployment.
Application Portability: Ensures consistency across development, testing, and production environments.

When to Use Virtual Machines:
Running Multiple OS Types: Useful for running applications that require different OS environments.
Legacy Application Support: Suitable for applications that require complete OS environments.
Strong Isolation: Best for scenarios that require enhanced security and resource isolation.

Pros and Cons

Docker:
Pros
Lightweight and fast
Limited security isolation
High portability
Easy to deploy and manage

Cons
Requires container orchestration tools for scaling
Kernel dependency on the host

Virtual Machines:
Pros
Strong isolation
High resource consumption
Supports multiple OS types

Cons
Slower startup times
Mature technology
Less portable than containers

Conclusion
Docker and Virtual Machines are powerful technologies, but they serve different purposes. Docker excels in scenarios where speed, portability, and resource efficiency are critical. In contrast, Virtual Machines are better suited for use cases requiring robust isolation, multiple OS environments, or support for legacy applications.

Knowing what the differences and strengths of each of them aids in choosing the best tools for our application needs. Often, a combination of both technologies can be used to build flexible and scalable infrastructures.

Learning DevOps: Setting Up a Local Workflow System Without the Cloud

Oni Toluwalope — Mon, 30 Dec 2024 19:33:54 +0000

DevOps has become an essential practice in modern software development and IT operations, with the cloud playing a significant role in enabling automation, scalability, and collaboration. However, cloud platforms often come with recurring costs and require an internet connection, which may not always be ideal for beginners or enthusiasts. Building your own personal setup provides an excellent alternative for learning DevOps concepts in a hands-on, cost-effective manner.

In this article, we’ll explore why a personal setup is valuable, what you need to set one up, and how to create your own DevOps practice setup without relying on the cloud. In the basic form of cloud computing, the main thing that makes the cloud different is accessing computing resources over the internet, so what if you could do the same on your PC without subscribing to a cloud provider.

Why Consider a Personal Setup Instead of the Cloud?

Advantages:
Cost Savings: No monthly subscription fees for cloud services. You can repurpose old hardware.
Offline Access: Learn and experiment even without an internet connection.
Privacy and Control: Complete ownership of your data and setup.
Hands-On Learning: Gain deeper insights into infrastructure, networking, and system administration by managing everything locally.

What you need to setup one

Essential Components of a Personal Setup

Hardware Requirements
PCs or Laptops: Ensure adequate RAM (8GB minimum), CPU (4 cores or more), and storage (SSD recommended).
Software and Tools
Virtualization Tools:
VirtualBox for lightweight virtualization needs.
VMware Workstation is another software for managing virtual machines.

Operating Systems
Linux distributions such as Ubuntu Server, CentOS, or Debian for hosting services.

Networking Tools and remote connection tools
OpenVPN or OpenSSH for remote access on your personal setup

Tools for DevOps Practice

Version Control: Git for managing code.
CI/CD Pipelines: Jenkins, GitLab CI, or GitHub Actions.
Configuration Management: Ansible, Chef, or Puppet.
Infrastructure as Code (IaC): Terraform for provisioning infrastructure.
Monitoring and Logging: Prometheus, Grafana, and ELK stack.
Containerization Platforms:
Docker for running and managing containers.
Kubernetes (K3s or MicroK8s) for orchestrating containerized applications.

Setting Up Your Personal Setup (the real deal)

Step 1: Define Your Goals

First things first, determine what you want to achieve with your personal setup. Common goals include:
Learning Linux system administration.
Hosting a website or web software locally.
Practicing containerization and orchestration with Docker and Kubernetes.
Building and deploying CI/CD pipelines.
Exploring networking concepts.

Step 2: Set Up the Setup

The setup here will be for a web software
Prepare the Hardware:
Choose a machine
Ensure adequate RAM (8GB minimum), CPU (4 cores or more), and storage
If you are on a windows enable virtualization from your BIOS
Install a hypervisor like VirtualBox.
Install any of the linux distros as base operating system on the hypervisor, there are 2 ways you can go about this.
a. Install from scratch
Use a stable Linux distribution from scratch such as Ubuntu Server or Debian.
Configure SSH for remote access.

b. Install via osboxes.org, which is what we will be doing. Preconfigured images are found here( I advise you go for this, since it is faster and you will be using it for practice)

Select the VM images dropdown
Click on virtual box images at the top, it will take you to page with a list of operating systems.
Go to https://www.osboxes.org/ and download your preferred linux distro.

Select ubuntu on the new page (choose the distro of your choice).
Download VirtualBox (VDI) image
I am downloading Ubuntu in this case.

The good thing about osboxes is that it is very fast to install

Download the your preferred version. I am downloading the 24.10 Oracular Oriole, and my hardware is a 64bit version (choose what matches your hardware)
Take note of the username and password in the info section

Save and extract the OS image to a folder.

On the virtual box interface, click on new, name your OS and select its type and version.

The folder is where my VMs are stored, you can use any foler of your choice.

Next allocate memory (RAM- 2 or 4GB) and processors(you can use 2), that will be available for the new virtual machine, then click next

Select "Use an existing virtual hard disk file" and browse to the folder that contains the extracted disk image. The disk must be present in this path must be present for the machine to work.

Click Finish

Your new virtual machine has been created.

Right click on the new OS and click on settings
You can reset the processor, storage you want to allocate to the virtual machine.

NETWORK
Go to network
Set the network to bridged adapter, this helps the virtual machine to get its own IP address as well as connect to the internet.

Start the VM to test, configure the settings as desired, and install anything you will need for the other machines as this will serve as the base for other machines, then shut it down after testing, to create clones.
the password is at info section of the osboxes website

Example tools you might need to install:
Docker and Kubernetes:
check if openssh is present
Install Docker for containerization.
Set up Kubernetes (K3s for lightweight clusters).
Ansible, Jenkins or GitLab CI for building and applications.
And many more softwares required

type ifconfig to check for the ip address
if 'ifconfig' is not installed, follow the installation process. use

 sudo apt install net-tools

run ifconfig again
I blocked you from seeing my ip address

check if openssh is present

sudo systemctl status ssh

If not, follow the following commands to install:

sudo apt update

Install the OpenSSH server package with the following command:

sudo apt install openssh-server

Check the status of the SSH service to ensure it's installed and running:

sudo systemctl status ssh

Start the SSH service:

sudo systemctl start ssh

Enable the SSH service to start on boot:

sudo systemctl enable ssh

Check the status of the SSH service again to ensure it's installed and running:

sudo systemctl status ssh

if it does not work as shown, follow the steps from the ssh section again.

shutdown the machine.

If you are on a windows machine, you can use mobaxterm to test and connect your VM via ssh.

CLONE THE VIRTUAL MACHINES

Right click on the newly created virtual machine and click clone.

name the first clone, I am naming mine "client machine" because that is what I intend to use as my local PC.
Ensure Generate new MAC addresses for all network adapters is selected

Click on linked clone(it saves space), then select finish

Repeat the cloning process for main-server, target-server1, target-server2. You will have something like at the end.
Remember: Generate new MAC addresses for all network adapters is selected

Sample setup
Start the client-machine and the target-server1.

let's change the name of the virtual machines from 'osboxes' to a preferred name. I am using 'clientmachine and targetserver1.

sudo nano /etc/hostname

sudo nano /etc/hosts

restart the machine and see the hostname change

open target-server1 machine and repeat the same steps, if you are familiar with ssh, login with ssh.

your output should look like this

Restart the machine and see the hostname change.

Next, follow this step-by-step guide to create a basic HTML website, test it locally, and then host it on your server with Nginx:
If you are familiar with creating basic HTML site and setting up nginx server, you can skip this part

create a basic html page on your client and test it.
Create the project folder:

mkdir my-website
cd my-website

Create the HTML file: Create an index.html file in the my-website folder:

<!DOCTYPE html>
<html lang="en">
<head>
<title>Page Title</title>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<style>
body {
  font-family: Arial, Helvetica, sans-serif;
}
</style>
</head>
<body>

<h1>My Website</h1>
<p>A website created by me.</p>

</body
</html>

example from w3schools

Enter firefox index.html in the cli to view the website

setup your server with this guide
Access the server: Use SSH to access your server: you can access it directly since it is on your computer, but I advise you use ssh, so you learn how to get into a real cloud environment

ssh user@your-server-ip

Update and install necessary packages:

Update the system:

sudo apt update && sudo apt upgrade -y

Install Nginx:

sudo apt install nginx -y

Verify Nginx installation: Start Nginx:

sudo systemctl start nginx

Check the status:

sudo systemctl status nginx

Step 3: Upload Your Website to the Server
Copy files to the server:
Use scp to upload your website files:

Ensure you run this command from the client machine when

scp -r my-website/* user@your-server-ip:/home/user/my-website/

Move files to the Nginx directory:
SSH into the server and move the website files to the default Nginx web directory:

sudo mv /home/user/my-website/* /var/www/html/

Step 4: Configure Nginx for Your Website
Edit the Nginx configuration:

Open the Nginx configuration file:

sudo nano /etc/nginx/sites-available/default

Update the server block to look like this:

server {
    listen 80;
    server_name your-domain.com;

    root /var/www/html;
    index index.html;

    location / {
        try_files $uri $uri/ =404;
    }
}

Test the Nginx configuration:

sudo nginx -t

Reload Nginx:

sudo systemctl reload nginx

Step 5: Verify the Website on the Server
Open your server’s IP address (e.g., http://your-server-ip) in a browser. You should see your website. Your server ip is the one that is shown when you use the ifconfig command.

If you have a domain, point it to your server’s IP address by updating the domain’s DNS settings. After DNS propagation, you can access the site using your domain.

Take note that the IP that is being mapped to this domain must be consistent with the ip of your server at any point in time, so be sure to update the ip accordingly whenever there is a restart, shutdown of server or any event that can trigger a change in the ip of your server machine.

If you do not want to go through the stress of constantly updating the DNS on a global accessibility scale, you can simply limit the availability and change to your local client machine by updating its hosts files in the /etc/hosts file

sudo nano /etc/hosts

Your output will be something like this

Run this system for other tools like ansible and kubernetes like you would have done on a cloud sever, using the main server and the target servers on local like you would on the cloud.

Linux User Creation Bash Script

Oni Toluwalope — Wed, 03 Jul 2024 17:26:19 +0000

This Bash script automates the creation of user accounts and group memberships in Ubuntu. It takes a text file as input, where each line specifies a username and the groups they should belong to (comma-separated). The script performs the following actions:

Reads the user and group information from the text file.
Creates user accounts if they don't already exist.
Creates groups if they don't already exist.
Adds users to the specified groups.
Generates random passwords for each user.
Sets permissions and ownership for user home directories.
Logs its actions in a file for reference.
This script simplifies user and group management, saving time and reducing the risk of errors compared to manual configuration.

Each User must have a personal group with the same group name as the username, this group name will not be written in the text file.
A user can have multiple groups, each group delimited by comma ","
Usernames and user groups are separated by semicolon ";"- Ignore whitespace
e.g.
Tolu;developer,tester,security

Code Breakdown.

#!/bin/bash

Helps the script to run in a bash shell once called.
This shebang! establishes a well-defined environment for the script

Defining Variables:

logfile="/var/log/user_management.log"

password_file="/var/secure/user_passwords.csv"

text_file=$1

The script utilizes variables to store crucial paths and user input. This enhances readability and maintainability. Here's a breakdown of the defined variables:

logfile: This variable holds the path for the log file where the script's actions are recorded. By default, it points to /var/log/user_management.log.
password_file: This variable stores the path to the password file. This file securely stores usernames and their corresponding randomly generated passwords. The default location is /var/secure/user_passwords.csv.
text_file: This variable captures the filename provided by the user as the first argument ($1). This file is expected to contain a list of usernames and their associated groups, separated by semicolons.

Input Validation:

The script ensures proper user input by performing validation. It checks if the user has provided the essential text file containing user and group information. Here's the code snippet for this validation:

   if [ -z "$text_file" ]; then
       echo "Error: Usage: $0 <name of text file>"
       exit 1
   fi

This block of code checks if the text_file variable is empty.

If the file is missing, an error message is displayed, informing the user of the correct usage (echo).
The script exits with an error code (exit 1) to indicate an issue with the input.

File Management: Creating Essential Files

Creating Directories:

The script employs the mkdir -p command to create the directory structure for the password file if it doesn't already exist. This ensures the script doesn't encounter errors due to missing directories. The -p flag in mkdir instructs it to create parent directories if necessary.

Creating Log and Password Files:

The touch command is used to create the log file ($logfile) and password file ($password_file). This establishes empty files for the script to record its actions and store passwords.

Setting Permissions:

The script prioritizes security by setting strict permissions (600) for the password file using chmod 600 "$password_file". This restricts access to the file, allowing only the owner to read and write to it. This prevents unauthorized access to sensitive password information.

Building Users and Groups: The actual automation section

generate_random_password

This function generates a secure and random password for each user.
```
  function generate_random_password() {
      local length=${1:-10}  # Default length of 10 characters
      tr -dc 'A-Za-z0-9!?%+=' < /dev/urandom | head -c "$length"
  }
```
- The function leverages /dev/urandom to access a cryptographically secure random number generator.
- It utilizes the tr command for character filtering. This ensures the password includes alphanumeric characters, special symbols (!?%+=), and avoids potential issues with spaces in passwords.
- Finally, head -c "$length" extracts the desired number of characters to form the random password.

log_message Function

This function simplifies logging by appending a timestamp and the script filename ($text_file) to the log file ($logfile). Here's the code:
```
  function log_message() {
      echo "$(date '+%Y-%m-%d %H:%M:%S') - $text_file" >> "$logfile"
  }
```
- The date command with the '+%Y-%m-%d %H:%M:%S' format generates a timestamp for each log message.
- The entire message is then appended to the log file using echo >>.

*create_user Function and User Creation *

This function handles user creation based on the user information in the text file. Here's a breakdown:
```
  function create_user() {
      local username="$1"
      local groups="$2"

      if getent passwd "$username" > /dev/null; 
  then
          log_message "User $username already exists"
      else
          useradd -m "$username"
          log_message "Created user $username"
      fi
  }
```
- It takes two arguments: username and groups (comma-separated list).
- The function first checks if the user already exists using getent passwd "$username" > /dev/null. If the command exits successfully, it means the user exists.
- If the user doesn't exist:
  - The script creates the user's home directory with useradd -m "$username". The -m flag instructs useradd to create a home directory for the user.
  - A success message regarding user creation is logged using log_message.
- Otherwise, a message indicating the user already exists is logged.

Group Management and User-Group Associations

The script iterates through each line in the text file, processing users and their assigned groups. Here's the process:

      while IFS=';' read -r username groups; do
          create_user "$username" "$groups"

          groups_array=($(echo $groups | tr "," "\n"))

          for group in "${groups_array[@]}";

         do

          if ! getent group "$group" > /dev/null; 

   then

  groupadd "$group"

  log_message "Group created $group"

  else

  log_message "Group $group already exists"

  fi

  usermod -aG "$group" "$username"

  log_message "Added user $username to group $group"

done

  - A `while` loop iterates through each line in the `text_file` 
      - Inside the loop:
          - The `create_user` function is called to create the user (already described earlier).
          - The comma-separated groups are split into an array (`groups_array`) using `tr` for easier processing.
          - Another loop iterates through each group in the `groups_array`.
              - It checks if the group exists using `getent group "$group" > /dev/null`.
`getend` is a command used to request file from database in the CLI

Security Home and Password Assignment:

security and finalizing the user setup:

Home Directory Permissions

The script prioritizes security by setting appropriate permissions for each user's home directory. Here's the code:
```
  chmod 700 "/home/$username"
  chown "$username:$username" "/home/$username"
  log_message "Set up home directory for user $username"
```
- The script restricts access to the user's home directory by setting permissions to 700 with chmod 700 "/home/$username". This grants read, write, and execute permissions only to the owner (the user).
- Ownership of the home directory is then transferred to the user with chown "$username:$username" "/home/$username". This ensures the user has full control over their home directory and its contents.
- A success message regarding home directory setup is logged.

Password Assignment

The script assigns a unique and secure password to each user. Here's the process:
```
  password=$(generate_random_password   # Generate 10-character password
  echo "$username:$password" | chpasswd
  echo "$username,$password" >> "$password_file"
  log_message "Set password for $username"
```
- It utilizes the generate_random_password function (described earlier) to create a 10-character random password for each user.
- The username and password are combined ("$username:$password") and used with chpasswd to set the password for the user.
- The script then stores both the username and the randomly generated password in the secure password file ("$password_file") for reference.
- Finally, a success message regarding password assignment is logged.

Bringing it All Together:

#!/bin/bash

#Log file and password location
logfile="/var/log/user_management.log"
password_file="/var/secure/user_passwords.csv"
text_file=$1

#check for file input
if [ -z "$text_file" ]
then
echo "Usage is: $0 <name of text file>"
exit 1
fi

#Create Log file and password files
mkdir -p /var/secure
touch $logfile $password_file
chmod 600 $password_file

#function to generate randompasswords
generate_random_password() {
  local length=${1:-10}
  tr -dc 'A-Za-z0-9!?%+=' < /dev/urandom | head -c $length
}

log_message() {
  echo "$(date '+%Y-%m-%d %H:%M:%S') - $text_file" >> $logfile
}

#FUNCTION TO CREATE USER
create_user() {
local username=$1
local groups=$2

if getent passwd "$username" > /dev/null;
then
 log_message "User $username already exists"
else
  useradd -m $username
  log_message "Created  user $username"
fi

#Adding user to groups
groups_array=($(echo $groups | tr "," "\n"))
for group in "${groups_array[@]}";
do
  if ! getent group "$group" > /dev/null; 
then
  groupadd "$group"
  log_message "Group created $group"
  else
  log_message "Group $group already exists"
  fi
  usermod -aG "$group" "$username"
  log_message "Added user $username to group $group"
done

chmod 700 /home/$username
chown $username:$username /home/$username
log_message "Set up home directory for user $username"

#Assigning Random password to users
password=$(generate_random_password 12)
echo "$username:$password" | chpasswd
echo "$username,$password" >> $password_file
log_message "Set password for $username"
}

while IFS=';' read -r username groups;
do
  create_user "$username" "$groups"
  done < "$text_file"

echo "User creation done." | tee -a $logfile

By combining these steps, the script automates the creation of user accounts, assigns them to designated groups, ensures secure home directory permissions, and provides a record of usernames and randomly generated passwords. This script streamlines user and group management, saving time and effort while promoting security best practices.

I am open to receiving comments with questions or suggestions that improves the script. Cheers!

Link to github repo

DEV Community: Oni Toluwalope

HOW A 30-MINUTE TASK TURNED INTO A 9-HOUR TROUBLESHOOTING NIGHTMARE

My First Mistake: I Didn't Check the Logs Immediately

My Second Mistake: I Misinterpreted What vm.max_map_count Meant

My Third Mistake: I Didn't Increase vm.max_map_count Sooner

My Fourth Mistake: I Overlooked Storage and Under-Allocated Resources

Key Takeaways from the Marathon Troubleshooting Session

Configuration Management: 12 factor, Factor III

DevOps #12Factor #ConfigurationManagement #AppSec #SecurityBestPractices #EnvironmentVariables

Dependency Management 101: Mastering the 12-Factor App's Factor II

Decoding the Shebang (#!) in Shell Scripts: A Deep Dive

ShellScripting #Bash #Linux #Unix #Programming #Automation #DevOps

Decoding the Shebang (#!) in Shell Scripts: A Deep Dive

ShellScripting #Bash #Linux #Unix #Programming #Automation #DevOps

Foundation of Consistency: The 12-Factor App's 'Codebase' Principle

Mastering Git: Understanding `git init` and `git clone` for Effective Version Control

Brief Intro to Operating Systems for Devops

Main Functions of an operating system

Other functions

Types of Operating Systems

Goals of an operating system

Why Operating Systems Matter in DevOps

Some Key Operating System Concepts for DevOps

How to Start Learning Operating Systems for DevOps

Docker vs Virtual Machines: What’s the Difference?

Learning DevOps: Setting Up a Local Workflow System Without the Cloud

Why Consider a Personal Setup Instead of the Cloud?

What you need to setup one

Setting Up Your Personal Setup (the real deal)

Step 1: Define Your Goals

Step 2: Set Up the Setup

check if openssh is present

CLONE THE VIRTUAL MACHINES

Linux User Creation Bash Script

File Management: Creating Essential Files

Building Users and Groups: The actual automation section

Security Home and Password Assignment:

My Second Mistake: I Misinterpreted What `vm.max_map_count` Meant

My Third Mistake: I Didn't Increase `vm.max_map_count` Sooner