DEV Community: Zoo Codes

The Cloud Exit Strategy: Architecting for Independence

Zoo Codes — Tue, 20 Jan 2026 11:00:59 +0000

We have normalized a dangerous standard in modern web development: paying hundreds of dollars a month for infrastructure before acquiring a single customer. We build distributed systems for simple problems. We rent our identity, our data, and our compute from vendors who profit from our complexity.

It is time to reject the complexity tax. It is time to return to the $5 VPS.

Here is the architecture of independence.

The Philosophy: The Modular Monolith

The industry tells you to split your application into microservices, serverless functions, and edge computing nodes. This is excellent advice for Netflix. It is terrible advice for you.

For 99% of SaaS applications, the correct architecture is a Modular Monolith running on a Linux server you control. This approach minimizes network latency, eliminates distributed systems failures, and—most importantly—gives you infinite runway.

This isn't just about saving money. It is about architectural sanity.

Layer 1: The Control Plane (Coolify / Dokploy)

The main reason developers flock to Vercel or Heroku is the "git push" deployment experience. Setting up Linux servers, configuring Nginx reverse proxies, and managing SSL certificates manually is tedious.

But we don't need to do that manually anymore. We can run our own Platform as a Service (PaaS).

The Choice: Coolify or Dokploy.
The Lesson: These tools act as a self-hosted overlay for your VPS. You connect your GitHub repository, and they handle the Docker builds, reverse proxies, and SSL certificates automatically.
Why: You get the Vercel experience (automated deployments, preview URLs) without the Vercel restrictions (bandwidth limits, function timeouts, vendor lock-in).

Layer 2: The Data Layer (SQLite + Litestream)

We have been gaslit into believing we need a distributed Postgres cluster for a standard CRUD application.

SQLite in WAL (Write-Ahead Logging) mode is not a toy; it is a production-grade database engine. Because it runs in-process with your application, query latency drops from 50ms (cloud roundtrip) to sub-0.1ms (memory access).

The Disaster Recovery Strategy:
The fear of SQLite is, "What if the server disk dies?"

The Solution: Litestream.
How it works: It hooks into SQLite’s WAL file and streams changes to commodity object storage (S3, Cloudflare R2, MinIO) in real-time. You achieve essentially zero RPO (Recovery Point Objective) for pennies a month.

Migration Path: When to Eject?

A common anxiety with this stack is scalability. "What happens when I grow?"

The truth is, SQLite handles reads effortlessly (it powers billions of mobile devices). The bottleneck is strictly concurrent writes. Because SQLite locks the database file during a write operation (even in WAL mode, though significantly optimized), you generally hit a ceiling at around 200–500 write transactions per second, depending on your hardware.

The Reality Check:
If you are processing 500 writes per second, you likely have tens of thousands of concurrent users. At that point, you are no longer a "bootstrapped project"—you are a successful business with revenue.

The Ejection Plan:

Vertical Scaling First: Before you leave SQLite, upgrade your $5 VPS to a $40 VPS. Faster NVMe drives and more CPU cores will extend SQLite's life significantly.
The Switch: If you truly saturate the drive I/O, you switch to Postgres. Because we are using standard SQL (via an ORM or query builder), this migration is often just changing a connection string and a driver.

Reaching the limits of SQLite is a "good problem." Do not optimize for it on Day 1.

Layer 3: Identity & Auth (Better-Auth / Authentik)

Identity is the stickiest layer of the stack. If you use Auth0 or Clerk, you are renting your users. If you stop paying, you lose your customers.

Option A: The Library Approach (Better-Auth)
For TypeScript/Node ecosystems, libraries like Better-Auth allow you to embed robust authentication (Social Logins, 2FA, Passkeys) directly into your monolith.

Pros: The user data lives in your database tables. Zero external dependency.
Cons: Tightly coupled to your codebase.

Option B: The Service Approach (Authentik)
For a language-agnostic solution (Go, Rust, Python), run Authentik as a container in your stack.

Pros: Provides a full SSO portal. Decouples auth from your application logic.
Cons: Uses more RAM than a library.

Layer 4: Observability (Prometheus + Grafana)

When your app crashes in the cloud, you check the dashboard they provide. When you run your own VPS, you need your own dashboard. Do not pay Datadog $500/month to monitor a $5 server.

We deploy a standard Prometheus + Grafana stack via Docker Compose.

Prometheus: Scrapes metrics from your app and your server (CPU, RAM, Disk I/O).
Grafana: Visualizes that data.
Loki (Optional): Aggregates logs.

This gives you deep visibility into your application's performance. You will understand your system better because you built the dashboard yourself.

The Application Layer: Language Agnostic

This architecture does not care what language you write in. The "blocks" of this stack—Docker, SQLite, Linux—are universal standards.

Go: Use Fiber or Echo for blazing fast binaries that sip memory.
Rust: Use Axum for type-safe, bulletproof performance.
Node/TS: Use Hono or FastAPI for rapid iteration.
Python: Use Django or FastAPI if data science is your core.

The interface between your code and the infrastructure is a Dockerfile. That is the only contract you need to fulfill.

Summary: The Cost of Freedom

Here is the bill for your production stack:

Service	Tool	Cost
Compute	Hetzner / DigitalOcean	$5.00 / mo
PaaS	Coolify (Self-hosted)	Free
Database	SQLite	Free
Backups	S3 / R2 (via Litestream)	~$0.02 / mo
Auth	Better-Auth	Free
Monitoring	Prometheus/Grafana	Free
Total		~$5.02 / mo

Conclusion

You don't need a venture capital budget to build scalable software. You need a Linux server and the courage to manage your own stack.

When you remove the bloated layers of managed services, you aren't just saving money—you are regaining control. You can host ten failed experiments on this server without paying a cent more. You have bought yourself time.

Stop renting your infrastructure. Own it.

If you have any questions or suggestions, please feel free to comment below or reach out to me on Twitter or LinkedIn. You can also check out my other articles on Dev.to. Thanks for reading!

Buy Me a Coffee

[Boost]

Zoo Codes — Sun, 18 Jan 2026 15:11:03 +0000

Unleashing DevOps with Automated VM Labs: Vagrant + Libvirt + Ansible 🚀

Zoo Codes ・ Jan 18

#virtualmachine #automation #linux #tutorial

Unleashing DevOps with Automated VM Labs: Vagrant + Libvirt + Ansible 🚀

Zoo Codes — Sun, 18 Jan 2026 15:10:20 +0000

Hello there! 👋

In this article, I'm going to share a method I've been using to create and manage virtual machine (VM) development and testing environments. By combining Vagrant, Libvirt, and Ansible, we can eliminate the inefficiencies of manual VM setup and finally say goodbye to the "it works on my machine" problem.

Objectives 🎯

In this walkthrough, we will cover:

Who this is for: Linux users, DevOps engineers, and developers looking to streamline their local lab setup.
What we'll learn: How to leverage Vagrant and Libvirt for high-performance local virtualization and Ansible for configuration management.
The Goal: To create complex, multi-node labs that are consistently configured, fast, and reproducible with single commands.

I. Introduction: The Problem with Manual VM Setup

Manual VM Drudgery: Traditional VM setup involves tedious GUI installers ("Next, Next, Next").
"Works on My Machine" Syndrome: Inconsistent environments lead to arguments and production issues.
The Promise of Automation: My goal is to create complex, multi-node labs that are consistently configured, fast, and reproducible with single commands.
The Solution: I'll show you how Vagrant, Libvirt, and Ansible can be the tools to achieve this automation.

II. The Dream Team: Vagrant & Libvirt Explained

A. Vagrant: The VM Choreographer

Definition: An open-source abstraction layer that simplifies the building and management of VM environments.

Key Benefits

Consistency: Eliminates "works on my machine" issues by using a shared blueprint.
Portability: Allows effortless sharing of entire development environments.
Isolation: Enables running multiple labs concurrently without conflicts.
Automation: Automates environment setup.
Disposability: Facilitates easy creation and destruction of labs for experimentation.
Production Parity: Mirrors real-world setups locally.

Core Concepts

Vagrantfile: The declarative script defining the environment.
Boxes: Pre-built VM images.
Providers: Underlying virtualization software (VirtualBox, VMware, Libvirt).
Provisioners: Tools (like Ansible) that configure VMs after they are running.

B. Libvirt: The Virtualization Maestro

Definition: An open-source virtualization management toolkit that provides a consistent API for interacting with various hypervisors (KVM/QEMU, Xen, VirtualBox).

Function: Acts as a translator and single management interface for different virtualization technologies.

Key Components

libvirtd: The background daemon managing VMs.
Libvirt API: The interface for developers.
virsh: Command-line interface for direct VM management.
virt-manager: Graphical interface for VM management.

Advantages on Linux

Native Integration: Deeply integrated with the Linux operating system.
KVM Leverage: Utilizes Kernel-based Virtual Machine (KVM) for near bare-metal performance.

C. The Unbeatable Combo: Vagrant + Libvirt

vagrant-libvirt Plugin: Enables Vagrant to use Libvirt for managing KVM/QEMU VMs.
Why it's Important:
- Superior Performance on Linux: I've found KVM offers significantly better speed and efficiency compared to VirtualBox.
- Enhanced Control: Provides greater control over complex setups.
- Freedom from Vendor Lock-in: Not tied to proprietary virtualization software.
Performance Differences (Reported):
- Faster startup times (10-30s vs. 15-45s for VirtualBox).
- Better network performance (utilizing virtio_net).
- Superior disk I/O.

III. Back to the Future: A Brief History of Automation

Vagrant's Genesis (2010):
- Started as Mitchell Hashimoto's personal project.
- Reached stable 1.0 release in 2012.
- Became the foundation for HashiCorp.
- Initially focused on VirtualBox, but its plugin architecture allowed for broader provider support.
Libvirt's Legacy (2005):
- Created to manage the growing virtualization landscape.
- Integrated KVM in 2006, solidifying its role.
- Evolved with tools like virt-manager and advanced networking.
- Reached 1.0.0 release in 2012.
Converging Power: Independent development led to synergy, with Vagrant providing orchestration and Libvirt providing performance.

IV. The Buzz & The Bumps: Current Community Opinions

A. The High Fives (Why People Love It)

Blazing Performance (on Linux): Libvirt/KVM consistently outperforms VirtualBox, especially for I/O-intensive tasks.
Native Linux Feel: KVM's integration with the Linux kernel provides a seamless experience.
Advanced Networking: Libvirt offers granular control over virtual networks, VLANs, and bridging for complex topologies.

B. The Head Scratchers (Where It Gets Tricky)

Steeper Learning Curve: Libvirt, virsh, and Linux networking concepts require more effort to master than VirtualBox's GUI.
Installation Quirks: Configuring vagrant-libvirt and its dependencies can be challenging due to missing packages or version conflicts.
Box Hunting: The ecosystem of pre-built Libvirt-specific boxes is smaller than for VirtualBox; conversion tools like vagrant-mutate may be needed.
Mac Users Beware: KVM is not native on macOS (especially Apple Silicon), making Libvirt less ideal; many macOS users opt for QEMU directly.

C. The Great VirtualBox Debate

VirtualBox offers cross-platform ease and a large box library.
Its Linux performance often requires significant tweaking (NFS for shared folders, virtio NICs) to match Libvirt/KVM.
VirtualBox's simplicity comes at a performance cost.

V. Roadblocks & Routers: Common Limitations ("Controversies")

The "Default Dilemma": VirtualBox dominates the pre-built box market, making Libvirt box availability a challenge.
Version Tango: Maintaining synchronized versions of Vagrant and the vagrant-libvirt plugin can be difficult.
Network Niggles: Configuring private IPs, ensuring synced folders, and resolving Libvirt network access issues can be frustrating for beginners.
The reload Hang-up: vagrant reload may not always apply Libvirt configuration changes; a full destroy and up cycle might be necessary.
One Provider at a Time (Mostly): Vagrant typically supports only one active provider (e.g., VirtualBox or Libvirt) unless explicitly managed.
IP Address Hide-and-Seek: Libvirt lacks a standard IP discovery mechanism, forcing the plugin to use less reliable methods (parsing dnsmasq logs, QEMU Guest Agent).

VI. Building a 3-Node Cluster with Vagrant + Libvirt 🛠️

This section provides a practical example of deploying multiple VMs that can communicate, managed by a single Vagrantfile.

The `Vagrantfile` Blueprint

Here is a complete Vagrantfile to set up a 3-node CentOS cluster with a private network.

# -*- mode: ruby -*-
# vi: set ft=ruby :

Vagrant.configure("2") do |config|
  # Global Libvirt Provider Settings
  config.vm.provider :libvirt do |libvirt|
    libvirt.driver = "kvm"
    libvirt.memory = 1024
    libvirt.cpus = 1
  end

  # Define 3 nodes: node1, node2, node3
  (1..3).each do |i|
    config.vm.define "node#{i}" do |node|
      node.vm.box = "generic/centos8"
      node.vm.hostname = "node#{i}.cluster.local"

      # Private Network (Libvirt will create a virtual bridge)
      node.vm.network "private_network", 
        ip: "192.168.121.1#{i}",
        libvirt__network_name: "vagrant-private-net"

      # Ansible Provisioner
      # automatically installs Ansible on the host if needed (if using 'ansible_local') 
      # or runs from the host machine to configure the guest.
      node.vm.provision "ansible" do |ansible|
        ansible.playbook = "playbook.yml"
        ansible.become = true
        ansible.limit = "all"
      end

      # Simple shell provisioning to update /etc/hosts for basic DNS
      node.vm.provision "shell", inline: <<-SHELL
        echo "192.168.121.10 node1.cluster.local node1" >> /etc/hosts
        echo "192.168.121.11 node2.cluster.local node2" >> /etc/hosts
        echo "192.168.121.12 node3.cluster.local node3" >> /etc/hosts
      SHELL
    end
  end
end

Key Blueprint Features:

Base Box: Uses generic/centos8, a reliable box for Libvirt.
Provider Config: Globally sets KVM driver, 1GB RAM, and 1 CPU per VM to save resources.
Loop Definition: Efficiently defines 3 nodes using a Ruby loop.
Networking: Assigns static IPs (.10, .11, .12) on a private network.
Shell Provisioner: rudimentary DNS via /etc/hosts so nodes can ping each other by name.

The Workflow

Install Prerequisites:
You'll need libvirt and qemu-kvm installed on your host system.
- Debian/Ubuntu: sudo apt install qemu-kvm libvirt-daemon-system libvirt-clients bridge-utils
- Fedora/RHEL: sudo dnf install @virtualization
- Note: Ensure your user is in the libvirt group (e.g., sudo usermod -aG libvirt $USER).
Install the Plugin:
```
vagrant plugin install vagrant-libvirt
```
Ensure Libvirt is Active: Check with systemctl status libvirtd.
Save the Vagrantfile: Copy the blueprint above into a file named Vagrantfile.
Launch the cluster:
```
vagrant up --provider=libvirt
```
SSH into a node:
```
vagrant ssh node1
```
Test connectivity:
```
ping node2.cluster.local
```

VII. Ansible: Your Automation Sidekick

This section details how to use Ansible for idempotent configuration management across the created cluster.

Why Layer Ansible: For repeatable configuration, a cornerstone of DevOps.
Prerequisites: Ansible installed, vagrant-libvirt plugin, and optionally qemu-guest-agent.

A. Two Paths to Provisioning

1. Vagrant's Built-in Ansible Provisioner (Easiest Integration)

Add an ansible block to the Vagrantfile, specifying the playbook.yml (as shown in the blueprint above).
Set ansible.become = true for root privileges.

Example playbook.yml:

Save this file in the same directory as your Vagrantfile.

---
- name: Configure Lab Nodes
  hosts: all
  become: true
  tasks:
    - name: Ensure pkg cache is up to date (CentOS/RHEL)
      dnf:
        update_cache: yes
      when: ansible_os_family == "RedHat"

    - name: Install essential tools
      package:
        name:
          - vim
          - curl
          - git
          - bash-completion
        state: present

    - name: Add a welcome message to /etc/motd
      copy:
        content: |
          ------------------------------------------
          Welcome to your Automated Vagrant Lab! 🚀
          Managed by Ansible.
          ------------------------------------------
        dest: /etc/motd

Run vagrant up or vagrant provision to apply changes.

2. External Ansible Control (More Flexibility)

After vagrant up, use vagrant ssh-config to get VM IPs and SSH keys.
Create a custom inventory.ini file.
Add your private key to the SSH agent: ssh-add.

Run playbooks directly:

ansible-playbook -i inventory.ini your_playbook.yml

Dynamic Inventory: For dynamic Libvirt environments, consider using community.libvirt for inventory management.

VIII. Crystal Ball Gazing: The Future of Automated VM Labs 🔮

Vagrant's Evolution & The Go Rewrite:
- HashiCorp has been working on porting Vagrant to Go (vagrant-go) to align better with their ecosystem (Terraform, Packer) and improve performance.
- While the "Vagrant 3.0" major release has been a long-term goal, the project continues to evolve with a focus on stability and maintaining the massive ecosystem of existing plugins.
- Expect continued bridging between local development and cloud-native workflows.
Libvirt's Ongoing Evolution:
- Hypervisor Enhancements: Continued improvements for QEMU interactions and support for diverse architectures like ARM64 (vital for running Linux VMs on Apple Silicon, though often via QEMU directly).
- Network Nirvana: Finer control over DNS, VLAN tagging, and high-performance networking improvements.
- Storage & Devices: Better NVMe support and snapshot handling.
The Broader Landscape of VM Orchestration:
- Hybrid & Edge: Seamless management of VMs from local machines to the cloud and edge devices.
- AI-Powered Automation: AI predicting issues, optimizing resources, and assisting with configuration.
- VMs & Containers Integration: Blurring lines for integrated management.
- Developer Delight: Tools focused on faster, cloud-native local development environments.
- IaC Everywhere: Infrastructure as Code becoming standard for all resources.

IX. Conclusion: Your Labs, Supercharged

I hope this guide helps you supercharge your labs! Vagrant, Libvirt, and Ansible offer a powerful, performant, and reproducible method for building virtual playgrounds, eliminating manual setup inefficiencies.

Embrace automation to reclaim time and innovate. Experience the magic of automated labs and share your adventures with the community!

If you have any questions or suggestions, please feel free to comment below or reach out to me on Twitter or LinkedIn. You can also check out my other articles on Dev.to. Thanks for reading!

Buy Me a Coffee

References 📚

[Boost]

Zoo Codes — Sun, 11 May 2025 09:23:51 +0000

Zoo Codes

May 10 '25

Crushing the Command Line: How I Used Amazon Q to Build a Smarter FastAPI Scaffolder

#devchallenge #awschallenge #ai #webdev

Comments

4 min read

Crushing the Command Line: How I Used Amazon Q to Build a Smarter FastAPI Scaffolder

Zoo Codes — Sat, 10 May 2025 15:06:15 +0000

This is a submission for the Amazon Q Developer "Quack The Code" Challenge: Crushing the Command Line.

What I Built

I created the FastAPI Scaffolder CLI – a zero-config generator for production-ready FastAPI apps. It solves common pain points:

🕒 Saves over an hour per project setup.
🛡️ Bakes in best practices (async DB pools, retry logic, secure defaults).
☁️ Generates Infrastructure as Code (Terraform/Docker for AWS/Local).

With one command, you get:

fastapi-scaffold create myapp \
    --db=postgresql \
    --broker=redis \
    --prod=aws

This generates a directory with all the files you need to get started, including a Dockerfile, docker-compose.yaml, and Terraform files for AWS deployment.

Key Benefits

✅ Simplifies database and configuration setup with auto-generated async SQLAlchemy/Motor integrations.
✅ Production-ready: Includes Terraform, Docker Compose, and monitoring tools.
✅ Flexible and fast: Designed to be less opinionated while still providing a solid foundation.

Existing scaffolding tools like the official FastAPI template often lack production readiness or are too rigid. This tool is built to be both flexible and easy to use.

Demo

The scaffolder supports:

🛢️ Multiple databases: PostgreSQL, MongoDB, SQLite.
💬 Message brokers: Redis, RabbitMQ.
📦 Deployment options: Minimal, Full (AWS ECS), Serverless.

Let me walk you through a typical workflow:

Install the CLI:

   uv pip install scaffold-fastapi

Run the scaffolder command:

   scaffold-fastapi customer-api --db=postgresql --broker=redis --stack=full

OR just pass the name of the app to see the default options:

   scaffold-fastapi customer-api

Generated files: The tool generates a complete project structure:

├── app/
│   ├── api/
│   │   └── v1/
│   ├── core/
│   ├── db/
│   ├── models/
│   └── main.py
├── tasks/
│   ├── celery_app.py
│   └── sample_tasks.py
├── infra/
│   ├── docker/
│   ├── terraform/
│   └── helm/
├── Dockerfile
└── docker-compose.yml

Run the app:

   cd customer-api
   docker-compose up

Access the API: Open your browser and navigate to http://localhost:8000/docs to see the auto-generated API documentation.
Run the Celery worker:

   docker-compose run --rm worker

Your FastAPI application is now running with:

PostgreSQL database with async connection pool
Redis for caching and message brokering
Celery worker for background tasks
Health check endpoints
API documentation at /api/v1/docs

When you're ready to deploy:

cd infra/terraform
terraform init
terraform apply

Project Structure

The scaffolder itself is organized into modular generators:

scaffold-fastapi/
├── scaffold_fastapi/
│   ├── generators/
│   │   ├── app.py       # FastAPI application files
│   │   ├── celery.py    # Celery task configuration
│   │   ├── docker.py    # Docker and docker-compose files
│   │   ├── env.py       # Environment variable files
│   │   └── terraform.py # AWS deployment files
│   └── __init__.py
├── cli.py               # CLI entrypoint
└── pyproject.toml
├── README.md
├── LICENSE

Code Repository

The complete code is available on GitHub: scaffold-fastapi. The repository includes detailed instructions and examples of the generated code.

The package is also available on PyPI: scaffold-fastapi.

How I Used Amazon Q Developer

Amazon Q Developer was the secret weapon that made this project possible. Here's how it helped:

Initial Concept and Planning

Defining the scope: Amazon Q helped identify the most valuable features for a FastAPI scaffolder, such as database support, message broker integration, and deployment options.
Designing the CLI interface: It suggested using Typer for a clean, intuitive command-line experience.
Planning the project structure: Amazon Q provided best practices for organizing FastAPI projects, including separating concerns and using async database connections.
Generating code snippets: It created boilerplate code for components like async database setup, Celery configuration, and Docker files.

Rapid Prototyping

Amazon Q accelerated development by:

Implementing the CLI: It generated the core CLI logic with proper argument handling and validation.
Creating generator modules: Each component had its own generator function, producing well-structured, idiomatic code.

Debugging and Refinement

Amazon Q was invaluable for:

Troubleshooting: It helped resolve Docker Compose networking issues.
Code optimization: Suggestions improved efficiency and maintainability.
Feature expansion: New features were implemented cleanly with its guidance.

Packaging and Distribution

Amazon Q assisted in:

Creating the pyproject.toml file: It generated a well-structured configuration for packaging and dependencies.
Setting up the CLI entry point: It provided the necessary boilerplate for Typer to work seamlessly with the package.
Writing documentation: It helped draft the README and usage instructions, ensuring clarity for users.
Creating a license file: It suggested using the MIT license, which is widely accepted and easy to understand.
Creating Github Actions CI/CD workflows: It generated a basic CI/CD pipeline for testing and deploying the scaffolder.
Creating a Dockerfile: It generated a Dockerfile for the CLI tool, making it easy to run in a containerized environment.

Find more about the packaging process in the packaging guide.md and packaging-and-dist.md

Lessons Learned

Start with clear requirements: The more specific your initial prompt, the better the results.
Iterative development: Break complex tasks into smaller chunks and refine incrementally.
Collaborative problem-solving: Treat Amazon Q as a pair programming partner, not just a code generator.

Impact and Future Plans

This scaffolder has already saved me countless hours. What used to take a full day now takes minutes. The generated code is robust and follows best practices.

Future Enhancements

Support for more databases like CockroachDB and DynamoDB.
Authentication templates for OAuth, JWT, and API keys.
Testing frameworks with pytest fixtures and example tests.
CI/CD templates for GitHub Actions and AWS CodePipeline.

Conclusion

Amazon Q Developer has transformed how I approach development. It's not just about writing code faster—it's about writing better code with fewer bugs and better architecture. This FastAPI scaffolder is just one example of how Amazon Q can help developers build smarter tools.

If you're building CLI tools or automation scripts, I highly recommend leveraging Amazon Q Developer. Start with a clear vision, break down the problem, and let Amazon Q help you implement the solution.

I hope you have enjoyed this article. If you have any questions, please feel free to reach out to me on Twitter. or LinkedIn. You can also check out my other articles on Dev.to. Thanks for reading!

Nested Virtualization using VMware Workstation, Esxi and Windows 10 on Linux - Part 1

Zoo Codes — Thu, 16 Jan 2025 07:31:01 +0000

Happy new year all 🎉, It's been a while since my last post. I am currently enrolled in a full-time course hence the lack of time. However, I'll try to do better in regards to getting a schedule in place.

Introduction

Recently, we were tasked with setting up:

Domain controller vm.
Web-server vm. both using Windows Server 2016 running on VMWare ESXi. Also:
Two windows 10 clients.
All of the above running on VMWare Workstation.

The following were additional requirements for the above project:

All the clients enrolled into the Domain.
A static site hosted on the web server vm.
The static site should be accessible on all clients through a custom URL.
Create 4 Organisational Units with two users each able to access the client VMS.
DNS server running on the domain controller.

Prerequisites

For anyone looking to replicate this project, the following are required:

A laptop/desktop with a 64-bit operating system (either Windows or Linux).
Atleast 16GB of RAM.
256GB of storage or more (preferrably an SSD).
Atleast 4 cores with virtualization enabled(check your BIOS options).
A browser.

Why not Use QEMU/KVM?

As a fellow Linux enthuasist (I use Arch btw), I can already hear the furious keyboard typing about using QEMU/KVM. I use it to host my Windows 11 VM with GPU Passthrough for gaming and 3D tasks. I might write about it eventually.

While I agree that QEMU/KVM is a great tool, I ran into a major issue after setting up the above project. This relating to networking, I couldn't for the life of me get the networking to work properly inside ESXi. I tried setting up a bridge, setting up a virtual network, and even setting up a virtual switch. Nothing seemed to work. I was about to give up when I decided to use VMware workstation.

Yes, its probably a skill issue on my end but I was able to get everything working with VMware workstation in a matter of hours.If you're able to replicate this setup, kindly let me know. I'd love to know how you did it. All that side, lets get to it.

Project Overview

The diagram below provides an overview of the project:

Here we're setting up a Type 2 Hypervisor (VMware Workstation) on a Linux Host then installing three Virtual Machines:

Type 1 HyperVisor(VMWare ESXi) -> inside we'll have 2 VMs: a. Domain Controller (Windows Server 2016) b. Web Server (Windows Server 2016)
Windows 10 client 1
Windows 10 client 2

The diagram below showcases a high-level view of the Virtual Machines:

Setting up VMware Workstation on Linux

VMware Workstation is a Type 2 Hypervisor that allows you to run multiple virtual machines on a single host. Following the acquisition by Broadcom, there was alot of uncertainty about the future of VMware Workstation. However, they now offer a free license for Personal use for Workstation.

To get started, follow these steps:

Download the VMware Workstation installer from the official VMware website. Check the appropriate version for Linux.
Log in to the Linux host with the user account that you plan to use with VMware Workstation.
Open a terminal interface. For more information, see Opening a command or shell prompt (1003892).
Change to root. For example:

su root

Note: The command that you use depends on your Linux distribution and configuration.

Change directories to the directory that contains the VMware Workstation bundle installer file. The default location is the Download directory.
Check if you have the /etc/init.d directory, if not create it before running the installer.

sudo mkdir /etc/init.d

Run the appropriate Workstation installer file for the host system.

For example:

sh VMware-workstation-Full-_xxxx-xxxx_._architecture_.bundle [--_option]_

Where:
- xxxx-xxxx is the version and build numbers
- architecture is i386 or x86_64
- _option_ is a command line option.

For further information, check the linux install docs.

Arch-Linux install

VMware relies on compiling it's own kernel modules, due to the bleeding edge nature of Arch -Linux, you may need to compile your own kernel modules. Incase you run into the following error:

This is a well-known issue with running VMware on Linux - compiling your own kernel modules. This isn't much of a problem for seasoned users but for others it might be a deal breaker. This repo will save you hours of searching google.

On Arch you can install VMware Workstation on Arch-Linux using the AUR or Chaotic AUR:

Install yay

  sudo pacman -S yay

Ensure all packages are up too date by running
```
yay
```
Install VMware Workstation

  yay -S vmware-workstation

Once completed launch using your application or run the following command in the terminal:

vmware

In-case you have an issue with the networking or get an error pop-up, start as root:

sudo vmware

Running ESXi on VMWare

Download the ESXi iso from the Broadcom website, the latest version is 8. However if your hardware is not compatible with the latest version, you can download version 7 here.

Installation is similar other VMs:

Create a New Virtual Machine.
Use the Typical Configuration.

Select the ESXi iso.

Choose the location to save the VM files.

Enter the Maximum disk size. In my case I used 60GB (Note the recommended is 142GB).

Lets Customize the Hardware options: i. Change the allocated memory to 16GB.(Adjust according to your host specifications). ii. Change the CPU cores to 12. iii. Change the Network Adapter to Custom. I'm using /dev/vmnet8. (This is important for connecting the client VMs)

Finish the setup and power on the VM.

Installing ESXi

Once Esxi powers on, you'll be greeted with the following screen:

Press Enter to continue, next, you'll be prompted to accept the EULA, press F11 to accept and continue.

The next screen will prompt you to select the disk to install ESXi on. Select the disk and press Enter.

You'll be prompted to enter a password for the root account. Enter the password and press Enter.

Finally you'll be prompted to press F11 confirm Installation of ESXi. Press F11 to install ESXi.

Once the installation is complete, you'll be prompted to remove the installation media. Press Enter to reboot the system.

Once the system reboots, you'll be greeted with the following screen once booting is complete:

Note: your IP address will be displayed on the screen. You can access the ESXi web interface by entering the IP address in your browser.
Due to SSL issues, you'll get a warning, click on advanced and proceed.

The login screen will be displayed, enter the root username and password you created during installation.

You'll be greeted with the following screen:

Configuring a new datastore

Before we can create new VMs, we need to create a new datastore in Esxi. But we need to first add a storage device to the VM in VMware Workstation.

Open VMware Workstation and select the ESXi VM.
Click on Edit virtual machine settings.
Click on Add.
Select Hard Disk and click Next.
Select SATA and click Next. If you have m.2 or NVMe drives, select the appropriate option.
Set the disk size to 60GB and click Next. (Note: You can adjust the size according to your needs).

Boot the VM and login to the ESXi web interface. Click on the storage tab > Adapters and you'll see the new storage device.

Now we can create a new datastore:

Click on the Datastores tab.
Click on New Datastore.

Select Create a new VMFS datastore and click Next.
Select the storage device and click Next.

Enter a name for the datastore and click Next.
Select Use full disk and click Next.

Confirm the settings and click Finish.

The new datastore will be created.

Conclusion

In this Part 1 of the series, we've covered:

Setting up VMware Workstation on Linux.
Installing ESXi on VMware Workstation.
Configuring a new datastore in ESXi.

In the next part, we'll cover:

Creating a new VM in ESXi.
Installing Windows Server 2016 on the new VM.
Configuring the Domain Controller and Web Server VMs.
Setting up the Windows 10 clients.
Enrolling the clients into the Domain.
Setting up the DNS server on the Domain Controller.

Stay tuned for Part 2. Feel free to ask any questions or provide feedback. I'll be happy to help. Happy learning!

References

28 Common Linux 🐧Questions And Answers 🖥️

Zoo Codes — Thu, 21 Nov 2024 20:33:41 +0000

Introduction

Welcome to the friendly world of Linux! If you’re curious about how this powerful operating system works, you’ve come to the right place. This article is packed with common questions and easy-to-understand answers that will help you get started with Linux.

Whether you’re a complete beginner or just looking to brush up on your skills, we’ve got you covered. From the basics to everyday commands, this guide will walk you through everything you need to know to feel comfortable using Linux. Let’s embark on this exciting journey together!

Common Linux Questions & Answers

1. What are the primary purposes and key features of an operating system, particularly Linux? The primary purpose of an operating system (OS) like Linux is to manage hardware resources and provide a user interface for interaction. Key features of Linux include multitasking, multi-user capabilities, security through user permissions, and a powerful command-line interface. Additionally, Linux is open-source, which allows for customization and community-driven development.

2. Describe the origins and development of the Linux operating system. Linux was created by Linus Torvalds in 1991 as a free and open-source alternative to UNIX. It started as a personal project but quickly gained popularity among developers and enthusiasts. Over the years, many distributions (distros) have been developed, each tailored for different needs, such as Ubuntu for ease of use and Fedora for cutting-edge features. The collaborative nature of its development has led to a robust and secure operating system.

3. What are the best practices for installing Fedora Linux? When installing Fedora, I've found that it's important to back up any important data, ensure that your hardware is compatible, and create a bootable USB drive with the Fedora ISO. During installation, I recommend choosing the custom partitioning option if you're familiar with disk management, and to keep the system updated regularly using the package manager to avoid security vulnerabilities.

4. Explain how to interact with a Linux system using a terminal and basic shell commands. Interacting with Linux through the terminal is a powerful way to perform tasks. Basic shell commands include ls to list files, cd to change directories, cp to copy files, and rm to remove files. Using the man command (e.g., man ls) can provide detailed information about any command, which is incredibly helpful for learning.

5. Describe the purpose of TCP/IP protocol and how it is configured in Linux. TCP/IP is the fundamental protocol suite for networking, allowing different devices to communicate over the internet. In Linux, you can configure TCP/IP settings through configuration files like /etc/sysconfig/network-scripts/ifcfg-eth0 for static IP addresses or use the nmcli command for managing network connections via NetworkManager.

6. How do you configure a network interface card (NIC) in Linux? To configure a NIC in Linux, I usually edit the network configuration files or use the ip command. For example, I can set a static IP by editing /etc/network/interfaces or using nmcli for NetworkManager-managed connections. After making changes, I restart the networking service using systemctl restart NetworkManager.

7. What are the key aspects of printer, log file, and user administration in Linux? Printer administration involves configuring printers using CUPS (Common Unix Printing System), where you can add, manage, and troubleshoot printers through a web interface. Log file management is crucial for system monitoring; log files located in /var/log/ provide insights into system performance and errors. User administration includes adding/removing users with commands like useradd and userdel, and managing permissions with chmod and chown.

8. How do you navigate the Linux directory structure using relative and absolute pathnames? Navigating the Linux directory structure can be done using both relative and absolute pathnames. An absolute pathname starts from the root directory (e.g., /home/user/documents), while a relative pathname starts from the current directory (e.g., documents if I’m already in /home/user). Using cd to change directories and pwd to print the current directory helps me keep track of where I am.

9. Explain the use of shell wildcards in specifying filenames in Linux. Shell wildcards are incredibly useful for matching filenames. For example, * matches any number of characters (e.g., *.txt matches all text files), while ? matches a single character (e.g., file?.txt matches file1.txt, file2.txt, etc.). Using wildcards saves time when working with multiple files.

10. Describe the major features of the BASH shell, including redirection and piping. Redirection allows me to send output from a command to a file using > (e.g., ls > filelist.txt) or read input from a file using <. Piping (|) lets me chain commands together, so I can use the output of one command as the input for another (e.g., ls | grep txt to list only text files).

11. How do you create and execute basic shell scripts in Linux? Creating a shell script is straightforward. I usually start by opening a text editor (like nano or vim), then I write my script starting with the shebang line #!/bin/bash to specify the interpreter. After writing the commands I want to execute, I save the file with a .sh extension. To execute the script, I need to make it executable using chmod +x script.sh, and then I can run it with ./script.sh.

12. What are the types of hardware typically found in server systems and how are they configured in Linux? Server systems typically include components like CPUs, RAM, storage drives (HDDs/SSDs), network interface cards (NICs), and sometimes specialized hardware like RAID controllers. In Linux, hardware configuration can be done through various commands like lspci to list PCI devices, lsblk to view block devices, and configuring storage through tools like fdisk or parted. Additionally, server settings can be managed through configuration files and system utilities.

13. Explain the RAID levels and their configurations in Linux. RAID (Redundant Array of Independent Disks) levels include RAID 0 (striping, no redundancy), RAID 1 (mirroring, redundancy), RAID 5 (striping with parity), and RAID 10 (combination of striping and mirroring). In Linux, I can configure RAID using the mdadm tool, which allows me to create and manage software RAID arrays. For example, to create a RAID 1 array, I'd use a command like mdadm --create /dev/md0 --level=1 --raid-devices=2 /dev/sda /dev/sdb.

14. Summarize the major steps necessary to boot a Linux system. Booting a Linux system generally involves several key steps:

BIOS/UEFI Initialization: The system firmware initializes hardware and performs a POST (Power-On Self-Test).
Bootloader Execution: The bootloader (like GRUB) is loaded from the boot sector, which then presents options for booting different kernels or operating systems.
Kernel Loading: The selected kernel is loaded into memory, and the initial RAM disk (initrd) is also loaded if necessary.
System Initialization: The kernel initializes system hardware and mounts the root filesystem.
Init System: The init system (like systemd) is started, which then launches system services and user sessions.

15.How do you configure the system to start and stop daemons at certain run levels or targets? In Linux, particularly with systemd, services (or daemons) are managed through units. To enable a service to start at boot, I can use systemctl enable service_name, and to disable it, systemctl disable service_name. To start or stop a service manually, I can usesystemctl start service_name or systemctl stop service_name. For configuring specific targets, I can use systemctl isolate target_nameto switch to a specific run level or target.

16.What is the Filesystem Hierarchy Standard in Linux? The Filesystem Hierarchy Standard (FHS) defines the directory structure and directory contents in Linux systems. It specifies that:

[ ] the root directory (/) is the top-level directory.
[ ] /bin for essential binaries,
[ ] /etc for configuration files,
[ ] /home for user home directories,
[ ] /var for variable data,
[ ] /usr for user programs and data.

This standardization helps ensure consistency across different Linux distributions.

17.Explain how to manage file and directory permissions in Linux. In Linux, file and directory permissions are managed using the chmod, chown, and chgrp commands. Permissions are represented in three categories: owner, group, and others, with read (r), write (w), and execute (x) permissions. For example, to give the owner read and write permissions, I would use chmod u+rw filename. To change the owner of a file, I would use chown new_owner filename, and to change the group, chgrp new_group filename.

18.Describe the types of device files in the /dev directory. In the /dev directory, there are two main types of device files: character device files and block device files. Character device files represent devices that transmit data one character at a time (e.g., /dev/tty for terminal devices), while block device files represent devices that read and write data in blocks (e.g., /dev/sda for hard drives). These files allow applications to interact with hardware devices through standard file operations.

19.How do you mount and unmount filesystems in Linux? To mount a filesystem in Linux, I typically use the mount command. For example, to mount a USB drive located at /dev/sdb1 to a directory called /mnt/usb, I would run:

sudo mount /dev/sdb1 /mnt/usb

To unmount it, I would use the umount command:

sudo umount /mnt/usb

It’s important to ensure that no processes are using the filesystem before unmounting it to avoid data loss.

20.How do you view processes and change their priorities in Linux? To view processes in Linux, I often use the ps command, which lists currently running processes. For a more dynamic view, top or htop provides real-time updates. To change a process's priority, I can use the nice command when starting a process (e.g., nice -n 10 command) or the renice command to change the priority of an already running process. Lowering the priority (increasing the niceness value) can be done with:

renice 10 -p PID

21.Explain the use of cron and at daemons for scheduling tasks in Linux. The cron daemon is used for scheduling recurring tasks at specified intervals. I can create a cron job by editing the crontab with crontab -e, where I specify the time and command to run. For example, to run a script every day at 5 AM, I'd add:

1 0 5 * * * /path/to/script.sh

The at command is used for scheduling one-time tasks. I can schedule a command to run at a specific time by using:

echo "command" | at 5 PM

22.Describe how to configure key infrastructure services like DHCP and DNS in Linux. To configure a DHCP server in Linux, I typically install the dhcp package and edit the configuration file located at /etc/dhcp/dhcpd.conf, where I define the subnet, range of IP addresses, and other options. After configuring, I start the DHCP service using systemctl start dhcpd. For DNS, I often use BIND (Berkeley Internet Name Domain). I install BIND, configure the zone files in /etc/bind/, and set up the named.conf file to define the zones. I then start the BIND service with systemctl start named.

23.Explain the process of configuring the Apache Web server for web services in Linux. To configure the Apache Web server, I first install it using the package manager (e.g., sudo apt install apache2 on Debian-based systems). After installation, I can edit the main configuration file located at /etc/apache2/apache2.conf or create virtual host files in /etc/apache2/sites-available/. To enable a virtual host, I use the a2ensite command, and then restart Apache with sudo systemctl restart apache2. I also ensure that the firewall allows HTTP/HTTPS traffic using ufw allow 'Apache Full'.

24.What are the different facets of Linux security and how can you increase the security of a Linux computer? Linux security encompasses user permissions, firewall configuration, regular updates, and system auditing. To increase security, I can enforce strong password policies, use tools like iptables or ufw for firewall settings, regularly apply security patches, and implement SELinux or AppArmor for additional access control. Additionally, I can minimize the number of running services and use SSH keys for secure remote access instead of passwords.

25.Describe effective troubleshooting practices for common hardware and software problems in Linux. Effective troubleshooting in Linux involves several steps:

Identify the Problem: Gather information about the issue, including error messages and logs.
Check Logs: Review system logs in /var/log/ (like syslog or dmesg) for relevant messages.
Use Diagnostic Tools: Utilize commands like ping, traceroute, and top to diagnose network issues and system performance.
Research: Look up error messages online or consult documentation.
Testing: Isolate the issue by testing hardware components or reverting recent changes.
Seek Help: If needed, reach out to forums or communities for assistance.

26.Explain the difference between UNIX and Linux operating systems. UNIX is a proprietary operating system developed in the 1970s, while Linux is an open-source clone of UNIX created by Linus Torvalds in the early 1990s.

27.What is Logical Volume Management (LVM) in Linux, and what are its advantages? Logical Volume Management (LVM) is a system for managing disk storage in Linux that allows for flexible allocation of storage space. With LVM, I can create logical volumes that can be resized dynamically, which is useful for managing storage needs over time. Advantages of LVM include:

Dynamic Resizing: I can easily increase or decrease the size of logical volumes as needed.
Snapshots: LVM allows me to create snapshots of volumes, which can be useful for backups or testing.
Pooling of Storage: Multiple physical disks can be combined into a single logical volume, simplifying storage management.
Improved Disk Utilization: LVM helps in efficiently managing disk space across multiple partitions.

28.Discuss the role of systemd in managing services and daemons in Linux. Systemd is a system and service manager for Linux operating systems that has become the default init system for many distributions. Its role includes:

Service Management: Systemd manages services (daemons) using unit files, allowing for easy starting, stopping, enabling, and disabling of services.
Parallel Startup: It can start services in parallel, which improves boot times compared to traditional init systems.
Dependency Management: Systemd handles service dependencies, ensuring that services start in the correct order based on their requirements.
Logging: It integrates with the journal system to provide logging capabilities, allowing for easy access to logs via the journalctl command.
Socket and Timer Activation: Systemd can start services on-demand when a socket is accessed or at scheduled times, optimizing resource usage.

Conclusion

Congratulations on exploring the basics of Linux! We hope this collection of common questions and answers has made your introduction to Linux a little easier and more enjoyable. Remember, every expert was once a beginner, so don’t be afraid to experiment and ask questions as you learn. With practice and curiosity, you’ll soon feel at home in the Linux environment. Keep exploring, and enjoy your journey into the world of open-source technology!

I hope you have enjoyed this article. If you have any questions, please feel free to reach out to me on Twitter. or LinkedIn. You can also check out my other articles on Dev.to. Thanks for reading!

Support me by buying me a coffee

My Secret to Staying Ahead of the Tech Curve: How daily.dev Has Been a Game-Changer

Zoo Codes — Mon, 02 Sep 2024 07:18:35 +0000

I'll be the first to admit it: as a developer, I've often felt overwhelmed by the sheer amount of tech news and trends out there. It's like trying to drink from a firehose! But about a year ago, I stumbled upon a tool that has completely transformed my daily routine: daily.dev[https://dly.to/CVhHhEWGKoN].

This amazing extension has not only saved me hours of time, but it's also helped me stay informed, focused, and ahead of the curve. I'm not exaggerating when I say it's been a total game-changer for me.

So, what makes daily.dev so special? For me, it's the personalized feed that's tailored to my interests and preferences. I no longer have to sift through endless lists of irrelevant articles or scroll through Twitter for hours. daily.dev's AI-powered feed gives me exactly what I need to know, when I need to know it.

But it's not just about saving time (although, let's be real, that's a huge bonus!). daily.dev has also helped me stay curious and excited about the tech industry. I love that I can quickly scan the latest news and updates, and then dive deeper into the topics that really interest me.

If you're a fellow developer looking to boost your productivity and stay up-to-date with the latest tech news, I highly recommend giving daily.dev a try. It's not just a tool – it's a community of like-minded folks who are passionate about tech and innovation.

So, what's your secret to staying ahead of the tech curve? Share your favorite tools and tips in the comments below!

From OVA to Qcow2: A Step-by-Step Guide to Unlocking Qemu's Power

Zoo Codes — Fri, 23 Aug 2024 05:45:15 +0000

Background

I was recently studying for and completed the CISCO CyberOps Associate certification. One of the labs required me to use a security VM in OVA format. I wanted to convert the OVA file to Qcow2 format to use it with Qemu, a popular open-source emulator and virtualizer. This wasn't necessary, I could have just used VirtualBox or VMware, but I wanted to explore the process of converting the file format and learn more about Qcow2.

Introduction

Virtual machines (VMs) have become an essential part of modern computing, allowing users to run multiple operating systems on a single physical machine. When working with VMs, it's not uncommon to encounter different file formats, each with its own set of features and compatibility issues. In this article, we'll explore the process of converting a VM in OVA format to Qcow2, a format compatible with Qemu.

What is OVA?

OVA (Open Virtualization Archive) is an open standard for packaging and distributing virtual machines. It's a single file that contains the VM's configuration, disk images, and other metadata. OVA files are widely supported by various virtualization platforms, including VMware, VirtualBox, and KVM.

What is Qcow2?

Qcow2 is a disk image format used by Qemu, a popular open-source emulator and virtualizer. Qcow2 is a more efficient and flexible format compared to traditional raw disk images, offering features like compression, encryption, and snapshotting.

Why Convert OVA to Qcow2?

There are several reasons why you might want to convert an OVA file to Qcow2:

Qemu compatibility: If you're using Qemu as your virtualization platform, you'll need to convert the OVA file to Qcow2 to ensure compatibility.
Performance optimization: Qcow2 offers better performance and compression compared to OVA, making it a more efficient choice for resource-intensive workloads.
Security: Qcow2 supports encryption and other security features, making it a more secure choice for sensitive workloads.

Converting OVA to Qcow2

Converting an OVA file to Qcow2 is a relatively straightforward process. You'll need to have the following tools installed:

OVA file: The VM in OVA format that you want to convert.
Qemu: The Qemu emulator and virtualizer.
qemu-img: A command-line tool for manipulating disk images.

Here are the steps to convert an OVA file to Qcow2:

Extract the OVA file: Use a tool like tar or 7zip to extract the contents of the OVA file. This will typically include a vmx file (VM configuration), a vmdk file (disk image), and other metadata. In my case I used 7zip to extract the OVA file in a relevant folder. The syntax is as follows:



7z x cyberops_workstation.ova -ocyberops_workstation

2.Identify the disk image: Locate the disk image file (usually a vmdk file).

3.Convert the VMDK file to Qcow2: Use the qemu-img command to convert the vmdk file to Qcow2. The syntax is as follows:



qemu-img convert -f vmdk -O qcow2 input.vmdk output.qcow2

NB: Note this may take some time depending on the size of the disk image. Also the siize of the image may increase or decrease depending on the conversion process. In my case the image size increased from 3.8GB to 8.3GB.

4.Create a new Qemu VM: Create a new Qemu VM using the qemu-system-x86_64 command (or similar, depending on your architecture). Specify the Qcow2 file as the disk image:



qemu-system-x86_64 -m 2048 -vnc :1 -hda output.qcow2

This will start the Qemu VM with 2048MB of RAM, VNC enabled on port 5900, and the Qcow2 file as the disk image. Replace 2048 with the desired amount of RAM, and :0 with the desired VNC port.

When you run this command, QEMU will:

Allocate 2 GB of memory to the virtual machine.
Start a VNC server listening on port 5901.
Use the output.qcow2 image file as the primary hard disk.
You can then connect to the virtual machine's graphical console using a VNC client, such as vinagre or tightvnc, by connecting to localhost:5901 or 127.0.0.1:5901.

Optionally, use Virt-Manager to create a new VM and use the Qcow2 file as the disk image.

Conclusion

Converting a VM from OVA to Qcow2 is a relatively simple process that can be accomplished using the qemu-img command. By following these steps, you can take advantage of Qemu's features and performance optimizations, while ensuring compatibility with your virtualization platform. Remember to always exercise caution when working with virtual machines, especially when dealing with sensitive workloads.

I hope you have enjoyed this article. If you have any questions, please feel free to reach out to me on Twitter. or LinkedIn. You can also check out my other articles on Dev.to. Thanks for reading!

Support me by buying me a coffee

Troubleshooting Common Issues in Kubernetes Deployments

Zoo Codes — Wed, 27 Mar 2024 16:55:04 +0000

Working with Kubernetes deployments can be challenging, and being prepared to tackle common issues is crucial for ensuring smooth application delivery and operations. This article aims to provide a comprehensive guide to troubleshooting some of the most frequently encountered problems when deploying applications on Kubernetes.

Introduction

Kubernetes has become the de facto standard for container orchestration, enabling organizations to deploy and manage applications at scale. However, as with any complex system, issues can arise during the deployment process or after an application is running. By understanding common issues and their corresponding troubleshooting steps, you can quickly identify and resolve problems, minimizing downtime and ensuring your applications run smoothly.

Issue 1: Deployment Failed Due to Invalid YAML Syntax

If you encounter an error that says "Deployment failed, reasons: [{}, '"invalid YAML syntax"']", it means that there is a problem with the YAML file you are trying to deploy.

To troubleshoot this issue, you can try the following steps:

Check the YAML file for syntax errors by running the kubectl validate command:



 kubectl validate -f /path/to/deployment.yaml

This command will validate the YAML file and report any syntax errors.

If there are syntax errors, fix them and reapply the YAML file using the kubectl apply -f command:



kubectl apply -f /path/to/deployment.yaml

Issue 2: Pods Stuck in Pending State

When pods are stuck in the Pending state, it indicates that Kubernetes is unable to schedule the pods onto any node in the cluster. This can happen due to various reasons, such as resource constraints, node failures, or network issues. To troubleshoot this issue, follow these steps:

Check the status of the nodes in the cluster by running the kubectl get nodes command:



kubectl get nodes

This command will show you the status of each node in the cluster, including whether they are Ready or NotReady.

Inspect the events related to the pending pods by running the kubectl describe pods command:



kubectl describe pods

Look for any error messages or warnings that might indicate why the pods are stuck in the Pending state.

Check the resource requests and limits specified in the pod's YAML file. Ensure that the pod's resource requirements are within the limits of the nodes in the cluster.
If the nodes are under heavy load, consider scaling your cluster by adding more nodes or adjusting the resource requests and limits for the pods.
If the issue persists, check for network connectivity problems between the nodes and the control plane. Ensure that the network plugins are correctly configured and that there are no firewall rules blocking communication.
If all else fails, you can try deleting and recreating the pods to force Kubernetes to reschedule them:



kubectl delete pod <pod-name>

Issue 3: Pods Are Not Running After Deploy

If you have deployed a new application and the pods are not running, it could be due to several reasons. Here are some steps to troubleshoot this issue:

Check the logs of the pods by running the kubectl logs command. This can provide information on the reason why the pods are not running: A ```yaml

kubectl logs


2. If the logs show any error messages, fix them and apply the changes using the `kubectl apply -f`command.
3. If the logs don't show any error messages, check for verbose logs by running the `kubectl logs` command with the -f flag:

```yaml


kubectl logs -n namespace pod-name -f

This will stream the logs in real-time, allowing you to observe any new error messages as they appear.

Verify if there are any ImagePullBackOff errors, which could indicate issues with pulling the container image. Check the image name and tag, ensure the image exists in the specified registry, and check network connectivity between the cluster and the registry.
Check for CrashLoopBackOff errors, which occur when a container crashes repeatedly after starting. Investigate the container logs for any error messages, verify environment variables and configuration files, and ensure that the application is compatible with the container runtime version.
If the pods are still not running, check the pod status and events by running the kubectl get pods and kubectl describe pods commands. Look for any error messages or warnings that might indicate why the pods are not running.

Issue 4: Deployment Rolls Back to Previous Version After Update

If a deployment rolls back to the previous version after an update, it could be due to several reasons. Here are some steps to troubleshoot this issue:

Check the deployment's rolling update strategy by examining the spec.template.spec.strategy.typefield in the deployment YAML file.
If the rolling update strategy is RollingUpdate, check the maxSurge and maxUnavailable fields to make sure they are set correctly. The maxSurge field specifies the maximum number of pods that can be created above the desired replica count during an update, while the maxUnavailable field specifies the maximum number of pods that can be unavailable during an update.




spec:
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0

If the rolling update strategy is BlueGreen, check the spec.template.spec.strategy.type field in the deployment YAML file. Ensure that the spec.template.spec.strategy.rollingUpdate field is set to BlueGreen and that the spec.template.spec.strategy.blueGreen field is configured correctly.



spec:
  strategy:
    type: BlueGreen
    blueGreen:
      activeService: active-service
      previewService: preview-service

If the deployment is still rolling back to the previous version, check the rollout history by running the kubectl rollout history deployment/<deployment-name> command. Look for any failed or paused updates and investigate the cause of the rollback.
If necessary, you can manually roll back to a previous version by running the kubectl rollout undo deployment <deployment-name> command.
Monitor the rollout status by running the kubectl rollout status deployment <deployment-name> command to ensure that the update completes successfully.
If the issue persists, check the container image tags and ensure that the correct image version is specified in the deployment YAML file. Verify that the image exists in the specified registry and that there are no network connectivity issues preventing the container from being pulled.
If all else fails, you can try deleting and recreating the deployment to force Kubernetes to start a new rollout:



kubectl delete deployment <deployment-name>
kubectl apply -f deployment.yaml

Issue 5: Services and Ingress Issues

If you're having trouble accessing your application through a service or ingress, you can try the following troubleshooting steps:

Verify the service or ingress configuration, including the port mappings, selectors, and endpoints.



# Service
apiVersion: v1
kind: Service
metadata:
  name: my-service
spec:
  selector:
    app: my-app
  ports:
    - port: 80
      targetPort: 8080



# Ingress
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: my-ingress
spec:
  rules:
  - http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: my-service
            port:
              number: 80

Check for any network policies or firewall rules that might be blocking traffic to the service or ingress.
Ensure that the appropriate ports are open and accessible from the client trying to access the application.
Verify that the DNS records are correctly configured to resolve the ingress hostname to the correct IP address.
Check the ingress controller logs for any error messages that might indicate issues with routing or load balancing.
If the service or ingress is still not working, check the pod logs for any error messages that might indicate issues with the application or container.

Issue 5: Resource Limits and Requests

If your pods are being evicted or not scheduled, it could be due to insufficient resource limits or requests. You can try the following troubleshooting steps:

Check the resource limits and requests defined in the deployment YAML file.



spec:
  containers:
  - name: my-container
    resources:
      limits:
        cpu: 500m
        memory: 512Mi
      requests:
        cpu: 250m
        memory: 256Mi

Ensure that the resource requests are within the limits of the nodes in the cluster. You can check the available resources on the nodes by running the kubectl describe nodes command.
If the pods are still not scheduled, consider adjusting the resource requests and limits for the pods or scaling your cluster by adding more nodes.
Monitor the resource usage of the pods by running the kubectl top pods command. Look for any pods that are consuming excessive resources and adjust the resource limits accordingly.

Issue 6: Persistent Volume Claims and Storage

If you're using persistent volumes or storage classes, you may encounter issues related to storage provisioning, mounting, and permissions. Here are some troubleshooting steps:

Verify that the persistent volume claim is bound to a persistent volume.



kubectl get pvc
kubectl get pv

Check the storage class configuration and ensure that it is correctly configured to provision the required storage.




apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: my-storage-class
provisioner: kubernetes.io/gce-pd
parameters:
  type: pd-standard

Check the pod logs for any error messages related to mounting the persistent volume. Verify that the volume mounts are correctly configured in the pod's YAML file.




spec:
  containers:
  - name: my-container
    volumeMounts:
    - name: my-volume
      mountPath: /data
  volumes:
  - name: my-volume
    persistentVolumeClaim:
      claimName: my-pvc

Ensure that the pod has the necessary permissions to access the persistent volume. Check the security context and service account configuration in the pod's YAML file.



spec:
  securityContext:
    runAsUser: 1000
    fsGroup: 2000
  serviceAccountName: my-service-account

If the persistent volume claims are still not working, check the storage provider logs for any error messages that might indicate issues with storage provisioning or mounting.

Conclusion

Troubleshooting common issues in Kubernetes deployments requires a systematic approach and a good understanding of the underlying components. By following the steps outlined in this guide, you can quickly identify and resolve issues related to YAML syntax errors, pod scheduling, application runtime, deployment updates, services and ingress, resource limits, and persistent volumes. Remember to monitor your applications and clusters regularly to proactively detect and address any potential issues before they impact your production environment.

Additional Resources and Best Practices

While this article covers some of the most common issues and troubleshooting steps, Kubernetes is a complex system, and there may be other issues that require further investigation. Here are some additional resources and best practices to help you stay informed and prevent issues:

Kubernetes Documentation: The official Kubernetes documentation is a comprehensive resource for understanding concepts, troubleshooting, and best practices.
Community Forums: Engage with the Kubernetes community on forums like Stack Overflow, Reddit, or the official Kubernetes Slack channel to seek help or share knowledge.
Kubernetes Best Practices: Follow best practices for resource management, configuration, testing, and monitoring to help prevent issues and ensure smooth deployments.
Linting Tools: Use linting tools like kubeval or kubectl-lint to validate your YAML files and catch errors before deploying.
Staging Environments: Implement proper testing and staging environments to catch issues before deploying to production.
Real-World Examples and Case Studies: Refer to real-world examples and case studies to learn from others' experiences and gain practical insights.

By staying informed, following best practices, and leveraging the community's knowledge, you can build resilient and reliable Kubernetes deployments that meet your organization's needs and deliver value to your users.

If you have any questions or feedback, please leave a comment below. You can also reach out to me on Twitter. or LinkedIn If you found this article helpful feel free to share it with others.

Buy me a coffee here.

Deployments and Load Balancing in Kubernetes

Zoo Codes — Mon, 18 Mar 2024 21:20:26 +0000

As modern web applications become increasingly complex and distributed, managing their deployment and operations can be a daunting task. Kubernetes, the popular open-source container orchestration platform, simplifies this process by providing a robust set of tools and abstractions for deploying, scaling, and managing applications in a reliable and scalable manner. In this article, we'll explore how Kubernetes can streamline the lifecycle of your web applications, from initial deployment to scaling, updates, monitoring, and more.

Introduction to Kubernetes Deployments

The foundation of running applications on Kubernetes is the deployment. A deployment defines how your application will be deployed and managed on the Kubernetes cluster. It specifies details like the number of replicated pods to run, the container images for the app, configuration data like environment variables, and update/rollout strategies.

Deployment Examples

For example, let's say you have a Python Flask web app built with the official python:3.9-slim image. Your deployment YAML could look like:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: flask-web-app
spec:
  replicas: 5
  selector:
    matchLabels:
      app: flask-web
  template:
    metadata:
      labels:
        app: flask-web
    spec:
      containers:
      - name: flask-app
        image: my-docker-repo/flask-web:v2
        ports:
        - containerPort: 5000
        env:
        - name: FLASK_ENV
          value: production

This instructs Kubernetes to create 5 replicated pods for the flask-web application. Each pod runs a container from the my-docker-repo/flask-web:v2 image with the container's port 5000 exposed. It also injects the FLASK_ENV=production environment variable.

You can define more advanced rollout configurations as well. For example:

spec:
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0

This rollout strategy does a rolling update - creating new pods slowly and taking old ones out of service until all are updated. The maxSurge setting allows provisioning 1 extra pod temporarily during updates, while maxUnavailable ensures all existing pods stay available.

Once you've defined your deployment YAML, you can apply it with kubectl apply -f deployment.yaml. Kubernetes will create/update the deployment, scheduling pods across nodes as needed.

Exposing Applications with Kubernetes Services

While a deployment runs your app's pods, a Kubernetes service exposes them for traffic from inside or outside the cluster. Common service types are:

ClusterIP (internal cluster access)
NodePort (external traffic on node ports 30000-32767)
LoadBalancer (provisions external cloud load balancer)

Service Examples

Continuing the Flask example, you could define a ClusterIP service like:

apiVersion: v1
kind: Service
metadata:
  name: flask-web  
spec:
  selector:
    app: flask-web
  ports:
  - port: 80
    targetPort: 5000

This internally exposes the flask-web pods on port 80, load balancing traffic to the container's port 5000. To access externally, you'd change the type to LoadBalancer.

You apply the service YAML via kubectl apply -f service.yaml. Kubernetes will configure the service and allocate a cluster IP (and external load balancer for LoadBalancer type).

Scaling and Updating Deployments

One of Kubernetes' key benefits is the ability to easily scale and update your deployments.

To scale up or down, you simply modify the replicas count and apply the updated deployment:

kubectl edit deployment/flask-web # Change replicas
kubectl apply -f deployment.yaml

Kubernetes will make sure additional pods get provisioned or terminated to match the desired replica count.

Updating to a new app version is just as straightforward - modify the container image tag in the deployment YAML and re-apply. Kubernetes will start a rolling update, gradually spinning up new pods on the updated image version and taking old ones out of service.

You can monitor rollout status via kubectl rollout status deployment/flask-web or get detailed rollout history with kubectl rollout history deployment/flask-web.

If you need to roll back to a previous version, that's just kubectl rollout undo deployment/flask-web.

Ingress and External Access

To expose services externally, you've seen you can use LoadBalancer services. There's also the concept of Ingress which acts as an external HTTP(S) load balancer.

An Ingress resource defines routing rules for external traffic to hit different services. For example:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress-web
spec:
  defaultBackend:
    service:
      name: default-service
      port:
        number: 80
  rules:
  - http:
      paths:
      - path: /flask
        pathType: Prefix
        backend:
          service:
            name: flask-web
            port:
              number: 80
      - path: /django
        pathType: Prefix
        backend:
          service:
            name: django-web
            port:
              number: 8000

This routes traffic to the /flask path to the flask-web service on port 80, /django to django-web service on 8000, and any other traffic to the default-service.

Monitoring and Logging

Finally, monitoring the health and logs of your deployments is crucial for ensuring the reliability and performance of your web applications. Kubernetes provides capabilities like:

Liveness and Readiness Probes to check app health
Resource Monitoring of CPU/Memory usage
Accessing Container Logs via kubectl logs
Integrating with Monitoring Services like Prometheus

For example, a liveness probe constantly checks if your app is running properly by hitting an endpoint like /healthz. If it fails, Kubernetes will restart the container.

You can also integrate various log shippers/aggregators to collect and analyze your application logs across all pods.

Conclusion

Kubernetes provides a rich set of tools and abstractions that simplify deploying, operating, scaling and monitoring modern web applications. By leveraging deployments, services, ingress, probes and other features effectively, you can run your apps in a highly available, self-healing environment. With Kubernetes, you can streamline the entire life-cycle of your web applications, from initial deployment to scaling, updates, monitoring, and beyond, enabling you to focus on building and improving your applications rather than worrying about the underlying infrastructure.

References:

Deploying Web Applications on Kubernetes: A Beginner's Guide

Zoo Codes — Fri, 16 Feb 2024 07:41:12 +0000

Happy new month everyone! Kubernetes has become the leading platform for deploying and managing containerized web applications. In this beginner's guide, we will walk through the key Kubernetes concepts and resources involved in deploying and updating web apps. We will also discuss the different deployment strategies that can be used to deploy web applications on Kubernetes.

Introduction

When deploying web applications on Kubernetes, two main resources are used:

Deployments - Used to manage multiple instances of a containerized web app.
Services - Expose deployments to external traffic.

We will cover how these resources work together to run and expose web apps. We will also discuss common deployment strategies like rolling updates, blue-green, and canary deployments.

By the end of this guide, you will understand:

The role of deployments and services
How to create and update deployments
Strategies like rolling updates to deploy new versions
How to expose apps externally using services

Let's get started! The diagram below gives a high-level overview of the steps we'll be going through.

Deployments in Kubernetes

A deployment controllers manages a set of identical pods (containers) that make up your application.

When you create a deployment, you specify the container image and the number of replicas (pods) you want to run. Kubernetes will spin up the pods and make sure the specified number are kept running.

For example, a simple deployment might create 3 replicas of an application container. If one of the pods goes down, Kubernetes will automatically re-create it to match the desired state.

Deployments allow you to easily scale up or down your application by changing the replica count. They also support rolling updates to push new versions of your application code.

Here is an example deployment manifest:



  apiVersion: apps/v1
  kind: Deployment
  metadata:
    name: my-webapp
  spec:
    replicas: 3
    selector:
      matchLabels:
        app: my-webapp
    template:
      metadata:
        labels:
          app: my-webapp
      spec:
        containers:
        - name: my-webapp
          image: nginx:1.7.9
          ports:
          - containerPort: 80

This will create a deployment named my-webapp that runs 3 replicas of the nginx:1.7.9 image exposed on port 80.

Services in Kubernetes

While deployments run containers inside pods, services expose your deployments so they can be accessed from outside the cluster.

There are different types of Kubernetes services. The main ones are:

ClusterIP - Exposes pods only within the cluster.
NodePort - Makes deployments accessible via node ports.
LoadBalancer - Provisions a cloud load balancer to route traffic.

For example, you may run a ClusterIP service to keep your app internal, and a NodePort service to allow external access.

Here is an example service manifest:



  apiVersion: v1
  kind: Service
  metadata:
    name: my-webapp-service
  spec:
    selector:
      app: my-webapp
    type: NodePort  
    ports:
      - port: 80
        targetPort: 80
        nodePort: 30007

This exposes our my-webapp deployment on port 30007 of every cluster node.

With deployments and services, you have the basic resources to run and expose containerized web applications on Kubernetes.

Next we will look at deployment strategies...

Deployment Strategies

There are several common strategies used to deploy new versions of your application:

Rolling Updates

The rolling update is the default strategy used if you apply a change to your Kubernetes deployment manifest.

It works by slowly replacing old pods with new ones a few at a time, rather than stopping everything at once. This allows changes to be applied without downtime.

The rollout pauses if too many pods become unavailable, ensuring availability. Rollbacks can also be performed.

Blue-Green Deployment

With blue-green deployments, instead of changing the existing deployment directly, you deploy the new version alongside the old version.

Once the new version is up and tested, you switch over by pointing the service to the new deployment. Traffic immediately shifts over to the new version.

This makes rollbacks as easy as switching the service back to the old deployment if issues arise.

Canary Deployment

With canary deployments, you slowly roll out a new version to a small subset of users to test it out before rolling it out to everybody.

This allows you to catch any issues with a new version early with minimal impact. Traffic is gradually shifted over time until all users have the new version.

Deploying a Web App on Kubernetes

Now that we have covered the key concepts, let's walk through a simple example of deploying a web application.

We will deploy the free open-source Cowsay app which displays a cow saying messages.

Step 1 - Create the deployment

First we create a deployment manifest cowsay.yaml:



  apiVersion: apps/v1
  kind: Deployment
  metadata:
    name: cowsay
  spec:
    replicas: 3
    selector:
      matchLabels:
        app: cowsay
    template:
      metadata:
        labels:
          app: cowsay
      spec:
        containers:
        - name: cowsay
          image: docker/getting-started
          ports:
          - containerPort: 80

This will deploy 3 replicas of the cowsay image exposed on port 80.

Step 2 - Create the service

Now we create a service manifest cowsay-service.yaml to expose the deployment:



  apiVersion: v1
  kind: Service
  metadata:
    name: cowsay-service
  spec:
    selector:
      app: cowsay
    ports:
      - port: 80
        targetPort: 80

This is a simple ClusterIP service to expose the cowsay internally.

Step 3 - Deploy the app

We can now deploy the app using kubectl:



  kubectl apply -f cowsay.yaml
  kubectl apply -f cowsay-service.yaml

That's it! The cowsay app is deployed and exposed on your Kubernetes cluster. Run the following commands to see deployment, services and pods that were created.



  kubectl get deployments
  kubectl get services
  kubectl get pods

From here you could update the deployment or expose it via a NodePort to make it externally accessible.

Summary

In this guide we covered:

Deployments - For running a number of identical pod replicas
Services - For exposing deployments internally and externally
Common deployment strategies like rolling updates and blue-green
Walkthrough of deploying a sample app on Kubernetes

We discussed how deployments can be used to manage multiple instances of a pod, how services can be used to expose web applications to the outside world, and the different deployment strategies that can be used to deploy web applications on Kubernetes. By understanding these concepts, you can begin to deploy your own web applications on Kubernetes.

This should give you a solid foundational understanding of running web applications on Kubernetes.

For more hands-on practice, I recommend checking out the Kubernetes documentation and trying out examples locally using minikube.

In the next article, we will explore how to scale and monitor web applications on Kubernetes. We will cover the different ways to scale your applications, and the tools and techniques required to monitor your applications. We will also discuss the different metrics that can be used to monitor your applications, and the different tools that can be used to visualize these metrics.

I hope you have enjoyed this article. If you have any questions, please feel free to reach out to me on Twitter. or LinkedIn. You can also check out my other articles on Dev.to. Thanks for reading!

Buy me a coffee here.

DEV Community: Zoo Codes

The Cloud Exit Strategy: Architecting for Independence

The Philosophy: The Modular Monolith

Layer 1: The Control Plane (Coolify / Dokploy)

Layer 2: The Data Layer (SQLite + Litestream)

Migration Path: When to Eject?

Layer 3: Identity & Auth (Better-Auth / Authentik)

Layer 4: Observability (Prometheus + Grafana)

The Application Layer: Language Agnostic

Summary: The Cost of Freedom

Conclusion

[Boost]

Unleashing DevOps with Automated VM Labs: Vagrant + Libvirt + Ansible 🚀

Zoo Codes ・ Jan 18

Unleashing DevOps with Automated VM Labs: Vagrant + Libvirt + Ansible 🚀

Objectives 🎯

I. Introduction: The Problem with Manual VM Setup

II. The Dream Team: Vagrant & Libvirt Explained

A. Vagrant: The VM Choreographer

Key Benefits

Core Concepts

B. Libvirt: The Virtualization Maestro

Key Components

Advantages on Linux

C. The Unbeatable Combo: Vagrant + Libvirt

III. Back to the Future: A Brief History of Automation

IV. The Buzz & The Bumps: Current Community Opinions

A. The High Fives (Why People Love It)

B. The Head Scratchers (Where It Gets Tricky)

C. The Great VirtualBox Debate

V. Roadblocks & Routers: Common Limitations ("Controversies")

VI. Building a 3-Node Cluster with Vagrant + Libvirt 🛠️

The Vagrantfile Blueprint

The Workflow

VII. Ansible: Your Automation Sidekick

A. Two Paths to Provisioning

1. Vagrant's Built-in Ansible Provisioner (Easiest Integration)

2. External Ansible Control (More Flexibility)

VIII. Crystal Ball Gazing: The Future of Automated VM Labs 🔮

IX. Conclusion: Your Labs, Supercharged

References 📚

[Boost]

Crushing the Command Line: How I Used Amazon Q to Build a Smarter FastAPI Scaffolder

Crushing the Command Line: How I Used Amazon Q to Build a Smarter FastAPI Scaffolder

What I Built

Key Benefits

Demo

Project Structure

Code Repository

How I Used Amazon Q Developer

Initial Concept and Planning

Rapid Prototyping

Debugging and Refinement

Packaging and Distribution

Lessons Learned

Impact and Future Plans

Future Enhancements

Conclusion

Nested Virtualization using VMware Workstation, Esxi and Windows 10 on Linux - Part 1

Introduction

Prerequisites

Why not Use QEMU/KVM?

Project Overview

Setting up VMware Workstation on Linux

Arch-Linux install

Running ESXi on VMWare

Installing ESXi

Configuring a new datastore

Conclusion

References

28 Common Linux 🐧Questions And Answers 🖥️

Introduction

Common Linux Questions & Answers

Conclusion

My Secret to Staying Ahead of the Tech Curve: How daily.dev Has Been a Game-Changer

From OVA to Qcow2: A Step-by-Step Guide to Unlocking Qemu's Power

Background

Introduction

What is OVA?

What is Qcow2?

The `Vagrantfile` Blueprint