DEV Community: Eden Jose

AWS, Kubernetes, and All Things DevOps

Eden Jose — Sat, 01 Jul 2023 03:45:10 +0000

I've been away for quite some time. Been busy with a lot of transitions-at home, at work, in life.

Been working on some projects but I've been posting them in Github as it's easy to manage and consolidate all projects into one single place. I will probably create some write-ups here in Dev.to on some of the interesting projects in my Github repositories, but in the meantime, here are some of the resources:

Eden JoseFollow

A cloud enthusiast, an IT Professional, and a problem-solver. I am either learning something new, running a mile, or planning my next 100 days.

Setting my bash prompt for WSL

Eden Jose — Sun, 09 Jan 2022 11:55:36 +0000

This is just a dump of my settings for bash prompt for my WSL terminal. I didn't like to show the whole path since it takes up the whole line already but I do want to see the directory where I'm currently at.

Appended this to the bottom of ~/.bashrc and then re-opened my terminal.

$ vim ~/.bashrc

export PS1="[\[\e]0;\u@\h: \W\]${debian_chroot:+($debian_chroot)}\[\033[01;32m\]\u@\h\[\033[00m\]:\[\033[01;34m\] \W\[\033[00m\]]$ "

Also, the desktop and dloads are aliases I added to the same file.

$ addalias

alias addalias='sudo vi /home/joseeden/.bashrc'
alias desktop='cd /mnt/c/Users/Eden\ Jose/Desktop'
alias dloads='cd /mnt/c/Users/Eden\ Jose/Downloads'

Attaching multiple EBS volumes to the same EC2 instance

Eden Jose — Mon, 06 Dec 2021 04:22:07 +0000

Just a short note. I created three EBS volumes to be used for my storage labs in Linux. I was going to attach them all at the same time to a single instance but the Attach volume is greyed out in the Actions tab.

However, if you mark them one by one, the Attach volume can be selected.

Summary

Multiple EBS volumes can be deleted at the same time
Multiple EBS volumes can be attached to the same instance, but only one at a time
Snapshots can be created for multiple volumes simultaneously

P.S.

AWS frequently changes their UI and functionalities so they may allow attaching volumes simultaneously in the future.
Also, this is just trivial matter since this probably won't matter if the volumes and instances are spun up in an automated way

EBS Volume stuck in "Attaching"

Eden Jose — Mon, 06 Dec 2021 00:29:17 +0000

Came across this error when I was doing labs for storage management in Linux. I was having issues on not being able to format the disks, so I finally decided,



"Ah, f*ck it. Imma just delete this volume and slap a new, 
fresh one in its place."

Detaching and Force Detaching

So there I went. Entered the AWS console and then to the EBS menu, selected my volume and deleted it. After detaching the volume, it still showed "In-use" volume state. Refreshed the menu a couple of times and it still showed the same. If the detachment was successful, it should have instead show "Available".

Another f*ck-it moment.



"Ah, f*ck it. Imma force delete this sh*t."

Selected the volume, hit Actions, and chose Force detach volume.
Note that it will prompt you to confirm by typing in detach on the prompt.

Afterwards, I deleted the volume so I can create a new one with the same device name.

Creating and Attaching the new EBS volume

Created a new EBS volume and gave it the same Name. Attached it to the same EC2 instance and reused the same device name /dev/sdc

EBS Menu vs. EC2 Instance Menu

From the EBS volume menu, the new volume is immediately transitioned to "In-use" state.

However, it still showing as Attaching from the EC2 instance menu. Refreshed the menu a couple of times and it still shows the same.

From the terminal, I also can't see the new /dev/xvdc.



eden@tst-rhel:~ $ sudo fdisk /dev/xvdc -l
fdisk: cannot open /dev/xvdc: Input/output error

Google Time

Went online and immediately found this helpful article from AWS themselves. It carefully explained what may have caused the issue. Breezed through it and found the culprit:

The block device driver didn't release the device name
If a user has initiated a forced detach of an Amazon EBS volume, the block device driver of the Amazon EC2 instance might not immediately release the device name for reuse. Attempting to use that device name when attaching a volume causes the volume to be stuck in the attaching state. You must either choose a different device name or reboot the instance.

Reboot

So I went and rebooted the instance. After it went back, i checked again the Storage tab in the Instance Summary panel. But now it showed a new error at the bottom. It also still shows the specific volume stuck in "attaching" state

Restart

Going back to the AWS articles, it suggested to do a restart.

If these steps don’t resolve the issue, or if you must use the device name that isn't working, try the following procedures:

Reboot the instance.

Stop and start the instance to migrate it to new underlying hardware. Keep in mind that instance store data is lost when you stop and start an instance. If your instance is instance store-backed or has instance store volumes containing data, the data is lost when you stop the instance.

Wait, hold up!

After re-reading the AWS article, I went back to the AWS Console to restart the instance. And what do you know. The new error is gone, the volume has also changed to "Attached" state.

Also can see it from the terminal



eden@tst-rhel:~ $ sudo fdisk /dev/xvdc -l
Disk /dev/xvdc: 100 GiB, 107374182400 bytes, 209715200 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

That's it.

References

Why is my Amazon EBS volume stuck in the "attaching" state?

Iptables and locking myself out

Eden Jose — Sun, 28 Nov 2021 20:28:29 +0000

What I'm using
Possible solutions
- SSH Client
- EC2 Instance Connect
- Session Manager
- EC2 Serial Console
- Detaching the root volume and attaching to a second instance
- EC2 Support
What really solved it!
Lessons Learned
- Decent iptables - A Working Rule Set for iptables per your requirements
  - Inbound filtering
  - Outbound filtering
  - Forward filtering:
References

Summary: Locked myself out via my own iptables policy. Other possible ways to connect are laid out but the one that worked is explained in the What really solved it section.

What I'm using

EC2 t2.xlarge instance with Elastic IP
I haven't installed any AWS-related agents before the incident
Security groups are set to allow only connections from either my IP or the security group itself
I didn't modify the default parameters set in the iptables-config file

I was playing around with DROP and REJECT on iptables and I'm using the "--protocol" as criteria to apply policy on. Since I've used udp and icmp already, I tried to drop tcp connections - bad idea.



$ sudo iptables -A INPUT --protocol tcp --in-intersection

After some searching, I learned that I'm basically fcked up. From a Stackoverflow discussion How to fix iptables if i have blocked all incoming and outgoing connections?

From comments, we've established this is on an Amazon AWS EC2 instance, and that you've locked yourself out from SSH access remotely. By using Amazon EC2, you're going to have a bit of a headache here. There's no real serial/console mode for access, nor anyone who can just 'fix' it, and by disabling all connections as you did, you've locked yourself out completely.

You don't really have much of a solution here but to destroy the EC2 instance and start over.

While this was helpful, note that this answer was posted in the Stackoverflow site last 2017 and there have been other ways created to connect to the instance. To see the options on how to connect to your EC2 instance, select your instance and click Connect.

Possible solutions

SSH Client

I've already used up SSH client when I blocked myself through iptables, so that leaves me with 3 more options.

EC2 Instance Connect

This didn't worked out as well since I needed to have EC2 Instance Connect installed on the machine before anything else. I tried to go the link provided in the error message Task 1: Configure network access to an instance. but it tackled more about security groups which isn't the one that's restricting me.

Session Manager

This would have been really useful because this uses SSH to connect to the instance. However, like EC2 Instance Connect, the Session Manager needs to be installed on the machine before anything else.

NOTE: Have this installed upon launching your instance or better yet define it in the startup script. Steps on how to install Session Manager in Linux can be read here.

EC2 Serial Console

This was a new one which I haven't seen last year during my review for the Associates certification exam. It required me to allow the account to use the EC2 Serial Console so I simply clicked Managed access and then ticked the Allow box and Update on the next page.

Going back to the previous EC2 instances landing page, selected my instance and hit Connect to see the EC2 Serial Console tab again. This time it showed another message - "This instance type is not supported for the EC2 serial console."

I went over the provided links and breezed through them. After allowing EC2 serial console access to the account, the instance type I'm using should also one of the supported instance type - Nitro. These are references you may check out:

Detaching the root volume and attaching to a second instance

This is actually an interesting method though I decided not to opt for this because I already found a solution - a really simpel one actually!
Keep scrolling down to read about it.

Also, I might create a separate article about this method.

EC2 Support

When all else fail, reach out to support. My free tier account has already expired years ago and I'm currently using AWS free credits I've obtained by joining the AWS Community Builders program. I am pretty sure that I'll have someone from their end answered some of my inquiries in the next 6 hours but I decided to not take the rouble of raising any support requests to them.

Since my focus at the moment is doing labs for the RHCSA topics and not on AWS, going through the linked documentations would be a rabbithole and would tend to be more complicated that just by simply deleting this instance and spinning up a fresh one.

Having said, I am considering to destroy it and start over. But this time, I'm having SSM Session Manager installed!

What really solved it!

I was almost on the verge of completely deleting the instance but then I remembered that iptables has a default config file called iptables-config, which contains two important parameters:



# Save current firewall rules on stop.
#   Value: yes|no,  default: no
# Saves all firewall rules to /etc/sysconfig/iptables if firewall gets stopped
# (e.g. on system shutdown).
IPTABLES_SAVE_ON_STOP="no"

# Save current firewall rules on restart.
#   Value: yes|no,  default: no
# Saves all firewall rules to /etc/sysconfig/iptables if firewall gets
# restarted.
IPTABLES_SAVE_ON_RESTART="no"

Basically what this two parameters meant is, if these two are set to "no", then rules will not persists when the machine is stopped or restarted. This means rules will be flushed out - yes!

So, after some restarting, I was able to log in again! All rules are also flushed out!

Didn't thought that a simple restart would save the day once again. Whew!



$ sudo  iptables -L -v
Chain INPUT (policy ACCEPT 379 packets, 47895 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain OUTPUT (policy ACCEPT 486 packets, 50206 bytes)
 pkts bytes target     prot opt in     out     source               destination

Lesson Learned

Going back to the original Stackoverflow discussion I've read earlier, it provided some important pointers which I thoguht to include here. I'm putting the original thread in the references section at the bottom.

Use the EC2 security group firewall instead. This is a little easier to configure, and it's already there without any additional configuration - it's part of the EC2 infrastructure where you have to permit ports to actually get to the EC2 instance. You also aren't going to lock yourself out as easily (though you can get locked out, it's trivial to fix it then because you just allow port 22 again in the rule set from the Amazon EC2 settings panel, provided you don't mess with iptables as well).
Use a decent iptables ruleset and don't log out from PuTTY on your EC2 until you are absolutely sure the rules you've put in place don't completely torpedo your access to the system.
Recommended to not mess with the default policies on a server, because it has some... evils... if not done correctly, and I usually only filter ingress traffic and FORWARD traffic, and permit Outgoing traffic because of time sync servers,

Decent iptables - A Working Rule Set for iptables per your requirements

You don't need to type lines that have a # at the beginning, those're just my comments explaining what each command does. Also, replace YOUR.IP.ADDRESS.HERE with your actual IP address, where it shows up below.

Inbound filtering



# Permit localhost to communicate with itself.
iptables -A INPUT -i lo -j ACCEPT
# Permit already established connection traffic and related traffic
iptables -A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
# Permit new SSH connections into the system from trusted IP address
iptables -A INPUT -p tcp --dport 22 -s YOUR.IP.ADDRESS.HERE -m conntrack --ctstate NEW -j ACCEPT
# Permit all other traffic from trusted IP Address
iptables -A INPUT -s YOUR.IP.ADDRESS.HERE -j ACCEPT
# Drop all other traffic
iptables -A INPUT -j DROP

Outbound filtering



# Allow Localhost to itself
iptables -A OUTPUT -i lo -j ACCEPT
# Allow RELATED,ESTABLISHED state traffic (related to Inbound for example)
iptables -A OUTPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
# Allow all other traffic to trusted IP address
iptables -A OUTPUT -d YOUR.IP.ADDRESS.HERE -j ACCEPT
# Drop all other unpermitted outbound traffic.
iptables -A OUTPUT -j DROP

Warning: This will block access to the update servers, time sync servers, etc. so ONLY filter on Outbound if you absolutely need to, otherwise don't do this section at all

Forward filtering:



# Drop FORWARD target traffic, we don't need it
iptables -A FORWARD -j DROP

NOTE: Unless you really need to restrict things like forwarding traffic to the Internet via a tunnel or VPN to your server as a 'proxy' to the 'net, you really don't need to mess with the FORWARD rulesets, so I would suggest not doing this because nothing else is really going to use this function or ever land in this rule set table

References

How to create cover images in dev.to with the title and author's name?

Eden Jose — Sun, 28 Nov 2021 18:58:07 +0000

I've been seeing articles here in dev.to with the banner/cover image showing the article's title, the publish date, and the author's name, plus the DEV logo a the right side, similar to the one below:

More examples:

I followed the instruction on this link but I'm having no luck in creating the "auto-generated banner from the article's title".

I did find this link about how "automated images" are created but I was wondering if there's other options that other people here dev.to is using?

EC2 Userdata script: Changing EC2 instance default username + run commands

Eden Jose — Sun, 28 Nov 2021 18:31:55 +0000

This is a short note on how to change the username that your EC2 instance uses upon startup. If you're using RHEL or CentOS, you probably see that the default username you see when you log in to your instance for the first time is ec2-user.

You can change this, plus run other commands during startup by using the script below and putting it on the User Data field during the creation of instance.

In this example, I'm setting the following:

default username: jeden
hostname: tst-rhel
format of the shell prompt

#cloud-config
system_info:
  default_user:
    name: eden

runcmd:
 - 'sudo hostnamectl set-hostname tstrhel'
 - 'sudo sed -i "s/localhost/localhost tstrhel/" /etc/hosts'
 - 'echo "PS1=\"[\u@\H: \W] $ \"" >> .bashrc'
 - 'sudo yum update -y'
 - 'sudo yum install -y vim nano wget bash-completion firewalld'
 - 'sudo yum install -y epel-release'
 - 'sudo yum install -y python3'
 - 'sudo yum install -y python3-pip python-devel'
 - 'sudo yum groupinstall -y "development tools"'
 - 'sudo pip install --upgrade pip'
 - 'sudo yum install -y policycoreutils-python-utils'

You may add more commands to be ran during startup by appending them in the runcmd section.

Note:

Some commands may not be ran during startup especially if the command requires that bootup process is done before it is executed.

A Shift in Thinking

Eden Jose — Thu, 25 Nov 2021 04:17:37 +0000

TLDR: And that I think is the real shift of thinking:
When you shift from being a learner to a problem-solver.

I realized that when you finished a course and learn some stuff and then work on actual projects, you seldom go back to the course and re-watch particular sections related to the issue you're troubleshooting.

Instead, you either a) go to your own notes or documentation to see if you've resolved it before (if you trust your documentation well enough), or you take the much easier and quicker route, b) you google it or search for sample use-cases in Youtube.

So, I think structured courses are really great when you're at the stage of discovery. But when you are trying to implement it to solve your own problems, you follow your own instinct to google it and hope someone has raised the same specific issues in Stackoverflow or hope that someone has made a video of it in Youtube.

Another thing, courses tend to get outdated easily especially when technologies grow at a really, really quick pace.

One day EC2 instances are limited to 20 instances per region, the next day its now based on number of vCPUs. Today you have a Zookeeper that handles all the syncing stuff, tomorrow it's removed entirely. Some changes may be small and some changes may be just a button moved in the UI, but add it up to all the other stuff that you're working on and they start to become significant.

One day you have weeks to play around on this fascinating new tech you just discovered, tomorrow you just have two weeks to explore it and come up with a bare minimum, usable solution.

I guess my point is, it took me almost 4 years (even now I'm still trying) to shift from a structured, college-tailored type of kind learning to the actual industry-setup of acquiring knowledge. I'm calling the college-type courses kind because you have a clear outline of what you need to follow, starting from the baby steps of firing up your first terminal, to eventually going through harder and more difficult stuff. You have a clear picture of what the problem is and what you need to accomplish first.

On the other hand, the actual industry doesn't lay out all the variables you need. It just presents a list of requirements, and it's up to you figure out what tools you need to use. It doesn't even give you a real problem, and it's up to you to come up with the questions and eventually, with the answers.

I guess this is what you might call ill-defined problems, where an almost perfect and unique solution oftentimes doesn't exist. During my stint at my previous job and in my current job, I've learned so much about how solutions are pieced together. You might have vendor's assistance or external professional services helping you out, but you'll come to realize that solving problems are not as straightforward as they seem. Some tasks need to be prioritized over others and some pressure from the business side may arise midway the sprint, which then prompts you to re-think which of the questions you have on your plate needs to be answered first.

So, the industry's way of learning tend to be a bit messy. You search what you need at the moment and you try it immediately on your test environment to see if it works. If it does, you try to incorporate piece-by-piece to your code. If it doesn't cause too much noise in the logs, if it passed all the unit tests, OAT, and such, and ultimately if everything seems to work smoothly, it might just find its way into production. But wait, here's the catch: you need to have all of it done in four sprints or less.

Now, this could introduce what you might call technical debt which is incurred when you prioritized a quick delivery over a perfect code.

Is this bad? Yes. No. I don't know. On one hand, you can't really perfect a code because it may depend on other factors that's out of your control such as the version of the platform you're running it on or the system updates that other teams are rolling out.

On the other, the quicker something fails, the quicker you can correct it. So it's as if we're always looking for ways to break the system so we can optimize it quickly.

Wait. Where was I?
Oh, yes, a shift of thinking. I guess I've laid out my take on how "learning-on-the-job" differs so much with "learning-on-demand".

So, should I stop taking courses and instead focus on projects and go straight to specifics?
If something's entirely new to you, you will need to build a solid foundation on it. That's what courses are for. They equip you with the basic tools that you (might) need to solve problems.

Along the way, you get to use these tools to answer your own questions. Other times, situations may require you to drop these tools altogether to adapt other tools better suited for the challenge at hand.

I am still confused.
Well, you're not alone. I still haven't figured all of it out yet. I still have a ton of courses in my laptop which I've yet to take. And since time is a limited commodity, I developed a strategy on how I could learn things better and apply them.

If my goal is to take a certification exam, I try to breeze through the course, do the labs, jot down notes, and prep up with some final practice tests. Once I passed the exam, I delete the course.

Yes. No hesitations. I know I wouldn't come back to it anymore and if I try to implement it on a project I'm working on, I'll just do a quick Google search on any error that I might encounter. I also get to pick just the specifics.

Just like when you're picking apples, you get the good ones and enjoy it later.

Bottom line is, I've equipped myself (I really hope I did) with the fundamentals during the discovery and review stage using the course as a tool. For the application stage, it's time to make use of my own tool: my Google-fu.

Currently, I'm prepping up to take RHCSA for next year. So by next year, I'm expecting to have any of the RHEL-related courses completely deleted from my hard drive.

By doing this, I'm forcing myself to think on my own instead of following the same exact steps that the instructor did. If I want to recall how I did a lab, I check my notes. If it's not on my notes, it's probably on Google.

Okay, I think I got it.
If you did, then you're already one step ahead of me and you did a great job. Again, I don't have all the answers yet, I don't even know what the specific questions are. But I do hope and expect to learn more as I go out and try new labs, fail on my code, tail the logs, test out solutions, and then document them. Along the way, I don't just learn. I solve problems.

And that I think is the real shift of thinking:
When you shift from being a learner to a problem-solver.

And when you've solved enough problems, you next shift to a much higher form of learning: passing it onto others.

Killing a defunct process

Eden Jose — Wed, 17 Nov 2021 22:40:32 +0000

What happened:
What this means
What to do
References

Encountered this when I was trying to install htop on one of my test-RHEL EC2 instances. I was following the steps in this link.

I've also learned that defunct processes are almost similar with zombie processes, but there's also some differences which you can read here.

What happened:

Two terminals opened, both connected to same instance
installed the package on second terminal while update was running on first

Got this message on second terminal when trying to install package and update simultaneously:

Running transaction check
Waiting for process with pid 5941 to finish.

Checking which process is that

$ ps -ef |grep 5941
root        5941    5939 18 06:00 pts/0    00:01:42 yum update -y
root       36173   36153  0 06:09 pts/2    00:00:00 grep --color=auto 5941

Since it was taking some time, I had to cancel the update running on the first terminal so I can install some packages first and then run the update again.

When I try install the packages again, it still returned same messaged about the same pid. Checking the processes again, it now showed "defunct"

$ ps -ef |grep 5941
root        5941    5939 18 06:00 pts/0    00:01:42 [yum] <defunct>

I tried killing by running kill -9 5941 but it still appear as a "defunct" process.

What this means

Based on one of the link I found online:

From your output we see a "defunct", which means the process has either completed its task or has been corrupted or killed, but its child processes are still running or these parent process is monitoring its child process. To kill this kind of process, kill -9 PID doesn't work. You can try to kill them with this command but it will show this again and again.

What to do

Determine which is the parent process of this defunct process and kill it. To know this run the command:

$ sudo ps -ef|grep defunct
UID          PID    PPID  C    
root        5941    5939  6 06:00 pts/0    00:01:42 [yum] <defunct>
eden       36210    6097  0 06:24 pts/2    00:00:00 grep --color=auto defunct

Then run the sigkill cmmand again but this time include both the PID and PPID.

$ sudo kill -9 5941 5939
$ sudo kill -9 36210 6097

Verify the defunct process is gone by ps -ef | grep defunct

References

Connecting via SSH from one EC2 instance to another

Eden Jose — Wed, 17 Nov 2021 15:58:36 +0000

Pre-requisites
Main Steps
If something goes wrong
References
Final Reminders!

This is another quicknotes which I tend to forget at times.

Pre-requisites

two or more Amazon EC2 instances
instances must be in same subnet and same availability zone

Main Steps

It actually only requires you to to generate an RSA key on each server. Assumption:

server-a is source (my server-a is RHEL)
server-b is destination (my server-b is Ubuntu)

On server-a, generate an rsa key by running:

ssh-keygen -t rsa
# You would need to enter a passphrase twice - recommended to have a passphrase

On you ~/.ssh folder, you should now see two id_rsa. One is a private key (something that's yours only) and a publc key(something you share). Note that you can rename your private and public key. Open the id_rsa.pub using vi editor and copy the contents. W

[eden@tst-rhel ~]$ cd .ssh/
[eden@tst-rhel .ssh]$ ll
total 16
-rw-------. 1 eden eden  799 Nov 17 22:03 authorized_keys
-rw-------. 1 eden eden 2655 Nov 17 21:44 id_rsa-rhel
-rw-r--r--. 1 eden eden  567 Nov 17 21:44 id_rsa-rhel.pub
-rw-r--r--. 1 eden eden  523 Nov 17 21:57 known_hosts

[eden@tst-rhel ~]$vi ~/.ssh/id_rsa.pub
ssh-rsa ************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************eden@tst-rhel

Now open another terminal and login to server-b. Go to the same ~/.ssh folder and open the authorized_keys file. Append the previously copied public key(from server-a).

[eden@tst-ubuntu .ssh]$ vi authorized_keys

ssh-rsa ************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************qwerty-keypair

# server-a public-key should be appended below.
ssh-rsa ************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************eden@tst-rhel

Trust should always be two-way. On server-b, generate the rsa keys (step 1), copy the public key (step 2), and then go back to server-a to append server-b's public key to the authorized_keys file (step 3).
From server-a, try to SSH to server-b.
From server-b, try to SSH to server-a.

If something goes wrong

Confirm that the IP you're using on the SSH command is still valid. The EC2 instances' public IP changes when stopped and started unless the instances are using Elastic IPs.
Check NACLs are set to default - they're normally unchanged
Ensure that they're on the same security group
Ensure that SSH through port 22 is allowed in the Inbound Rules section of the security group.
Try creating another destination EC2 instance in the same public subnet/Availability zone
You may also create another VPC and instances inside that VPC

References

These are some links that I find to be useful. You may find some others

Final Reminders!

Yes, even if you've done the steps a couple of times in the past, you might still forget how to do it. This is the why of this notes
Having said, always good to document.
Never ever share your Private key. That's yours and yours alone!
It is recommended to use key-based authentication instead of password-based authentication
You may try to search easier methods of connecting by using passwords but remember, passwords can be brute-forced!
Lastly, enjoy!

Get Virtual! with Python Virtual Environments

Eden Jose — Mon, 11 Oct 2021 16:41:23 +0000

These are used to isolate environments and resolve dependency conflict resolution.

This ensure projects have isolated siloes where they live and have their own dependency tree that don't interfere with one another.

Another thing, feel free to jump to sections that you're only interested in!

Python Dependencies
Installing Libraries
Using a requirements.txt
Using env
Using venv
Using virtualenvwrapper
Errors
- mkvirtualenv: command not found
- virtualenvwrapper Command '' not found, but can be installed with ...
References

Python Dependencies

You will often use 3rd-party libraries - non-default libraries which doesn't come along with Python.
You'll have to import and install them before they can be used.

normally installed using pip or easy_install
libraries are pulled from the pupi.org or Python Package Index which as an enormous index of libraries

Installing Libraries

To install a library, as example, django

pip install django

To install a specific version of django,

pip install django==2.2.12

To check the version of Django installed, you can run any of the commands below:

django --version
python -m django --version

You can also view the versions of all installed packages, including Django,

pip freeze
python -m pip freeze

To save all of these data (versions of each modules) to be reused or processed later, you can forward them to a file.

pip freeze > module-versions.txt

To upgrade to a newer version

pip install --upgrade django

To uninstall it, don't delete the folder, instead

pip uninstall django

Another example: installing the "site" library

import site
print(site.getsitepackages())

Using a requirements.txt

Another way to install multiple third parties with just a single command is to put them into a single requirements.txt file and use pip to to do a bulk-install.

(Note that it's recommended to install modules for a project in a virtual environment. Read below to learn more about this.)

To install from a requirements file

pip install -r requirements.txt

Using env

Let's take django as our example again. We recently installed django version 2.2.12. If we are to install a new version, v3.0, then it will override and uninstall the v2.2.12.

Some projects need specific versions of a library and this is where virtual environments come into play. Virtual environments exist to isolate projects and their dependencies from one another.

To create a virtual environment, we can use venv

python -m venv my-project-1

After you run this, a folder for the virtual environment will be created.

$ ll
total 0
drwxr-xr-x 1 Eden Jose 197610 0 Sep 14 12:16 my-project-1/

$ ll my-project-1/
total 5
drwxr-xr-x 1 Eden Jose 197610   0 Sep 14 12:16 Include/
drwxr-xr-x 1 Eden Jose 197610   0 Sep 14 12:16 Lib/
-rw-r--r-- 1 Eden Jose 197610 121 Sep 14 12:16 pyvenv.cfg
drwxr-xr-x 1 Eden Jose 197610   0 Sep 14 12:16 Scripts/

To activate a virtual environment, run the "activate" script inside the Scripts folder of the project. As an example,

$ source ./my-project-1/Scripts/activate
$
(my-project-1) 
Eden Jose@EdenJose MINGW64 ~/Desktop/Git/5-Virtual-Envs

Notice that the name of the virtual environment can now be seen at the prompt. To exit out of the virtual environment, simply run

deactivate

Using venv

Another way to create virtual environments is through virtual env.
Note that virtual env doesn't ship alongside your Python installation. To install virtual env,

pip install virtualenv

This take a similar step to create a new environment and activate it,

virtualenv my-project-2

# you can also use
python -m virtualenv my-project-2

You can create the virtual environment with a different Python installation by using the "-p" flag

# Check the python installations you currently have
$ where python
C:\Users\Eden Jose\AppData\Local\Programs\Python\Python39\python.exe
C:\Users\Eden Jose\AppData\Local\Programs\Python\Python38\python.exe

# You can choose from this two and create a virtual env with that version
python -m virtualenv -p

Using virtualenvwrapper

This is what I am using in labs and even at work. Treat this section as standalone, and can be setup even without the venv or env

This is another virtualenv library which wraps up some useful management functionality for virtualenv. One feature of this is it manages a single location of all your projects.

Unlike virtualenv and env where the project folder is created on your current working directory, virtualenvwrapper creates a folder is the user's home directory.

virtualenvwrapper maintains this folder where all your environment folders are created by default. You can setup your own directory where all the virtual environments folder will be created by creating the variable WORKON_HOME.

I had some problems when I was trying this one. As a solution, I just uninstalled any existing virtualenvwrapper installed on my system and do a fresh install. This

To do a fresh install,

joseeden@EdenJose:~$ sudo pip uninstall virtualenvwrapper
joseeden@EdenJose:~$ sudo pip install virtualenvwrapper

Next, append this to your .bashrc file


# Change the WORKON_HOME path to your directory where you want all your virtual environment folders created
export WORKON_HOME='/mnt/c/Users/Eden Jose/Desktop/Git/5-Virtual-Envs'
export VIRTUALENVWRAPPER_PYTHON='/usr/bin/python3'
source /usr/local/bin/virtualenvwrapper.sh

To create a virtual environment

mkvirtualenv <name>

# Sample
joseeden@EdenJose:~$ mkvirtualenv project-a
created virtual environment CPython3.8.10.final.0-64 in 21319ms...
(project-a) joseeden@EdenJose:~$

# Note that to create another virtual envionment, exit out of the previous virtual environment by running "deactivate"

joseeden@EdenJose:~$ mkvirtualenv project-b
created virtual environment CPython3.8.10.final.0-64 in 21319ms...
(project-b) joseeden@EdenJose:~$

To see all your virtual environments,

joseeden@EdenJose:~$ workon
project-a
project-b

To switch between virtual environments, you can simply run

joseeden@EdenJose:~$ workon project-b
(project-b) joseeden@EdenJose:~$

To exit out of a virtual environment

(project-b) joseeden@EdenJose:~$ deactivate
joseeden@EdenJose:~$

ERRORS

mkvirtualenv: command not found

If you get an error "command not found", this might mean virtualenvwrapper was not properly installed.

You can simply re-do the installation.
I had issues also when trying the virtualenvwrapper on Git Bash in VSCode so I decided to run the commands below in WSL.

sudo pip uninstall virtualenv -y 
sudo pip uninstall virtualenvwrapper -y
sudo pip install virtualenv
sudo pip install virtualenvwrapper
echo "WORKON_HOME='/mnt/c/Users/Eden Jose/Desktop/Git/5-Virtual-Envs'" >> ~/.bashrc
echo "source `which virtualenvwrapper.sh`" >> ~/.bashrc
. ~/.bashrc

virtualenvwrapper Command '' not found, but can be installed with ...

When you create a virtual environment using the mkvirtualenv command, you might see this error message

joseeden@EdenJose:/mnt/c/Users/Eden Jose$ mkvirtualenv project-a

Command '' not found, but can be installed with:

sudo apt install mailutils-mh  # version 1:3.7-2.1, or
sudo apt install meshio-tools  # version 4.0.4-1
sudo apt install mmh           # version 0.4-2
sudo apt install nmh           # version 1.7.1-6
sudo apt install termtris      # version 1.3-1

To resolve this, add the lins below (in this order) to your .bashrc file

export VIRTUALENVWRAPPER_PYTHON='/usr/bin/python3'
source /usr/local/bin/virtualenvwrapper.sh

## Note that the path depends on where your Python is installed. To check
which python
which python3

References

Kept running "source .bashrc" every time I open WSL

Eden Jose — Mon, 11 Oct 2021 14:51:22 +0000

I was playing around and testing some stuff in Linux when i suddenly did a tweak which changed how my WSL terminal opens up. This is what my terminal looks like - notice it doesn't the distinct green + blue colors.

After some searching I found two articles (links at the bottom) which describes the "sequence" which Unix follows when reading files. As summary, here's the files (in order):

/etc/profile
~/.bash_profile
~/.bash_login 4 ~/.profile

I added the function for .bashrc at the top of the .bash_profile:



## Loads .bashrc
if [[ -f ~/.bashrc ]] ; then
        . ~/.bashrc
fi

Closed terminal and reopened it
Voila! It now shows the green and blue colors.

Adding this great explanation from the reference link:

By default, Terminal starts the shell via /usr/bin/login, which makes the shell a login shell. On every platform (not just Mac OS X) bash does not use .bashrc for login shells (only /etc/profile and the first of .bash_profile, .bash_login, .profile that exists and is readable). This is why "put source ~/.bashrc in your .bash_profile" is standard advice