DEV Community: Eric Gitonga

Wrong Selection in R

Eric Gitonga — Mon, 28 Jan 2019 23:40:03 +0000

While reading the section on data frames in The Book of R, I got a result that was wrong from that in the exercise. Wrong because I had left out a comma in the code. Before I would just have corrected it and moved on, but this time around I wanted to know why it is I got that wrong result.

The data frame I was working with is

The code I typed was mydata[mydata$sex=="F"] This resulted in the following output:

Not quite what I expected. I then typed out just the test itself mydata$sex=="F" and got the following vector of logicals.

Looking at it, and comparing it to the output I got, things started making sense. The two columns I got back from my original code coincided with the logical TRUE values in the vector from the second code. What I had done by leaving out the comma was make a selection of columns instead of a selection of rows meeting the test criteria.

With this knowledge, I typed out the proper code mydata[mydata$sex=="F",] to get

The result I expected. Forcing myself to slow down and actually understand why code gives the output it does is doing wonders for my learning progress. Had I not understood from earlier reading and exercises how row and column selection coupled with logical values actually works, this would have been a much more difficult issue to figure out. Onwards with learning R!

Resources:

The Book of R: https://nostarch.com/bookofr

Excluding Diagonal Elements from a Matrix in R

Eric Gitonga — Thu, 10 Jan 2019 18:12:15 +0000

I am going through The Book of R by Tilman M. Davies as an introduction into things Data Science. During one of the exercises (4.1 part e on page 64), I came across this problem:

By extracting the diagonal elements of the logical matrix created in (c), use any to confirm there are no TRUE entries.

The said matrix is a 10×10 identity matrix. How does one do that extraction and testing? I tried everything that had been taught from the book thus far and could not find a way to do it. Off to Google I went in search of a solution.

From various sources, I got to learn of the lower.tri() and upper.tri() functions. Those seemed like they could do what I wanted. Only hitch is, I can only extract either the lower or upper half depending on the function used.

However, I have learned that you can use the c() function to combine different items into one vector. That means I can use the two functions to extract the portions I need, then combine the results, then do the required test.

For simplicity, I shall use a 3×3 identity matrix to demonstrate. First, let us form it using the diag() function and assign it to the variable t.

Next we use each of the lower.tri() and upper.tri() functions to identify the lower and upper halves of t.

In the arguments of both functions, we set diag = FALSE since we want to make sure the diagonals are left out. If we did not have that argument set, we would capture the ones in the diagonal, contrary to our goal.

Now we can extract each of those halves of zeros from t.

Since we want to store these as FALSE, we test to see if they are equal to TRUE.

Now we can use the c() function to combine these two halves into one vector s.

Finally, we can carry out the operation requested in the exercise.

This confirms that extracting the diagonals of an identity matrix will result in there being no logical TRUE values left in the matrix.

There is probably a simpler way to do this. I also have a niggling thought that there is a way to extract the diagonal ones and leave the zeros in the matrix, then test the matrix using any(), but I have yet to find out how. This is one of those beginner solutions that a certain period from now will be proof of how much my knowledge of things R will have grown. If you are aware of how to do this in a simpler, more straight forward way, please do share in the comments.

Edit: And yes, there is a simpler way! I just needed to be patient and read on a few more pages of the book to page 69 which introduces the function which. Using it saves so much typing!

Basically, the command selects elements in the vector that are to be extracted based on a test given. To extract some and leave others, you append a hyphen to which resulting in anything that passes the test being deleted.

My original question still remains though. Is there a way to extract the unwanted elements and still remain with a matrix? A possible solution would probably be to replace the ones with zeros then just test the matrix as a whole. Again, a little patience and reading on and paying attention (as this solution came after reading page 68) is all I needed. That led me the following:

Here I used the == relational operator to identify and replace the ones in the matrix with zeros. The subsequent steps are as before; convert the zeros to FALSE then run the matrix through any(), giving us the answer expected.

This is all very exciting. There is lots to learn and discover about programming with R, and data science by extension!

Resources:

The Book of R: https://nostarch.com/bookofr

Command Line Interface Photographs Conversion

Eric Gitonga — Thu, 28 Jun 2018 16:17:05 +0000

There is something magical about working on the command line. Sure, many find it super esoteric and arcane, but I love it. The world of Windows had really spoiled me big time, getting me used to the GUI for everything. Sure, there’s the command prompt, and for those that want more power, the aptly named PowerShell, but that still did not move me. After switching to Linux, I am reacquainting myself with the command line interface (CLI), and I find my love for it growing more and more.

In my regular photography duties, I realized I needed to generate thumbnails of images for a certain project. I am still exploring the ins and outs of digiKam, RawTherapee and GIMP, but something in me kept telling me that for this particular task, a CLI tool was better suited. So off I went searching. I had heard of and used ImageMagick before, but initial searching led me to GraphicsMagick, which I decided to try out.

After installing it, I ran GraphicsMagick with the command

gm help

to see what options it had. Among the options, the one that was of interest to me was the convert option. On searching for help about it from the man page, I found an example that fit my needs. The full command I needed was

gm convert -size 85x85 "infile.jpg" -resize 85x85 +profile "*" "outfile.jpg"

On testing it with an image, it worked just fine, giving me the output I expected.

Next up was trying to figure out how to apply this command to an entire directory of images. I went googling around for instructions on how to do that. After a bit of searching, I came across a listing that I modified to suit my specific needs as shown below:

for i in *.jpg;
do
filename=${i%.*}
gm convert -size 85x85 "$filename.jpg" -resize 85x85 +profile "*" "$#filename.jpg"
done

I saved this to a file gmthumbs (since what I wanted to do was to convert full size images to thumbnails) and used

chmod 777 gmthumbs

to make it executable.

I ran the command ./gmthumbs in a test directory I had set up with a number of images.

I didn’t get the results I expected, since only the first file was converted and given a new name (0filename.jpg) as shown in the following image:

I played around with the output variable name, having just the pound sign in the output file name, adding the dollar sign to it, then, eventually, removing the pound sign but keeping the dollar sign. That last option, listed below, gave me the results I needed.

for i in *.jpg;
do
filename=${i%.*}
gm convert -size 85x85 "$filename.jpg" -resize 85x85 +profile "*" "$filename.jpg"
done

My only problem now was the original files getting overwritten by the converted files. What I needed it to do was write the files to a new directory. I added the command to make an output directory and modified the output filename variable to have it moved to that new output folder as shown in the listing below

mkdir output;
for i in *.jpg;
do
filename=${i%.*}
gm convert -size 85x85 "$filename.jpg" -resize 85x85 +profile "*" "output/$filename.jpg"
done

On running the command and then asking for a listing of files, I found that the original files were still intact, and a new directory output had been created.

Going into that directory, I found the converted files in place.

Success! Now I can create thumbnails of images in any folder quickly and easily using this nifty script.

Before I close, I want to point out feh, which is a command line utility that I used to view the images. Running it as is without any options loads the images in a folder to a window that you can scroll through. Very handy for when I want to just view images I am working in from the command line without having to load a bulky GUI!

Now off to explore what other magical CLI tools I can incorporate into my workflow.

Resources:

ImageMagick: https://www.imagemagick.org/script/index.php

GraphicsMagick: http://www.graphicsmagick.org/

Feh: https://feh.finalrewind.org/

The Eternal Self Constraining Pseudo Student

Eric Gitonga — Thu, 14 Jun 2018 15:01:53 +0000

Should I read more? More manuals? More books? Should I watch more? More tutorials? More lessons? Should I listen more? More podcasts? Or should I just dive in and do? Iron out the kinks as I go along? Pick up material as and when it is needed? Am I hiding behind a fear of the unknown, using this “need to learn more” as a scapegoat?

Today, when pushed against the wall, I realized that this, indeed, might just be the case. RawTherapee is the software I am now using to edit photographs. Having gotten so used to Lightroom, this new software is a beast to get to grips with. Everything really is, the first time round. And the lure of the familiar triples that percieved difficulty. The event that forced me to learn about features of RawTherapee that I need for my day to day editing was a godsent. It forced me to look through the software to figure out how to carry out certain editing operations that I have been otherwise lax to learn.

Beyond this, what else I’m I putting off doing because the difficulty of actually learning it is so uncomfortable I would rather spend hours reading/watching/listening to learning media? It is now time to take stock. Time to work on that personal photography project. Time to start building. Time to bring out the drawing/painting material and create. Time to join an open source project and contribute. Time to put away the literature/videos/podcasts (other than for use as reference when stuck) and just do.

Sketch from photograph Libre by Tony Rojas on Unsplash

RawTherapee

Rediscovering the Joy of Linux*

Eric Gitonga — Sat, 09 Jun 2018 17:43:37 +0000

Recently I installed Ubuntu 18.04 to dual boot on a troublesome laptop alongside Windows 10. Everything went relatively fine eventually. That is after a rocky start trying to get my files on Windows backed up. I had to do so via a LiveCD before installing Ubuntu. That took quite some time. The hard disk is probably faulty, so it has to be replaced soon. Once it was up and running, it was time to install the various software applications I need.

I found that Cinelerra isn't natively supported by Ubuntu, so I need to figure out how to get that sorted. It is such a powerful tool from what I have read of it (and with such beautiful results as shown in the sample of user videos on its site) that I have to grok it.

I was surprised to see that Blender is supported. I really shouldn't have, seeing as it is open source, but I guess using it in Windows had made me forget its open source origins.

Handbrake is also supported! That, I don't think I'd have known. I got to know of it one time while looking for a proper way to convert video files in Windows.

I had forgotten that GIMP doesn't natively open raw files, so in reading around for a solution, I found RawTherapee. I haven't yet played with it though. Once I do, be sure I will write down my thoughts on it. I had known of Ufraw as a raw file editor, but on researching, what I found suggested RawTherapee was a much better option.

digiKam gets to replace Lightroom. From what little I've read about it, it is quite the powerful application. There's the digiKam Recipes Book that I am getting soon to help me learn the software. As its author Dmitri Popov points out, actually working on projects, albeit recipe format, is a much better and more effective way to learn software than to go through each menu option individually. And it is high time I got out of tutorial purgatory!**

Some other applications I installed are Chrome, Libre Writer/Calc, Filezilla, Git, Geany (after reading about it on Dev.to as a possible alternative to Windows' Notepad++) and Audacity.

There are other applications I had gotten used to in Windows that I need linux alternatives for. As and when the need to use them arises, I shall document the process of getting them. I do know that I need to set up wireless printing on the Canon PIXMA for both documents and discs. Initial research shows I can get that done. I just need to sit down and go into the details of it.

I want to do Whatsapp/Instagram/other apps sans mobile phone. It was an adventure trying to get that sorted. Prior to switching to Linux, I had come across a site that had an application called Franz which enables one to have several messaging apps under one roof. I thought I had struck gold until I downloaded and installed it. Instead of it being an emulator that allowed me to run the native Whatsapp app, it turns out it was just a wrapper that used web Whatsapp, which meant I needed an actual phone to scan the barcode to allow it to run my account. Clearly, that was not going to work for me.

On searching some more, I found a site called TechViola that listed seven applications that would allow me to run Whatsapp natively. Some of the applications are in alpha stage, some in beta, some paid, some free. I tried the easiest at first glance (Andy OS), only to realise it is a Windows only option. Next was via an extension on Chrome (ARC Welder), but that failed to launch the apps. Next, I tried Anbox and rain into issues installing it. They do say it is in alpha mode, so I shouldn't have been too surprised. Only thing is, getting it off was a problem. The installation hang, thus making it impossible to do anything else with apt. Each time I did, I was asked to run the command

sudo dpkg --configure -a

to clean things up. Which then just hang again. Running

sudo apt update
sudo apt upgrade

led nowhere either. In researching, I realised I needed to delete the two anbox modules in /usr/src, since each time I tried to update, apt would look there to see what sources lay therein to work on. I ended up deleting the two offending sources manually. Then ran

sudo dpkg --purge -a

to clean up any pending config issues. I then ran

sudo apt update

to clear up the configuration. Finally

sudo apt upgrade

executed smoothly without any errors.

In the process, I got reacquanted with a lock file and the commands ps, grep and kill. Each time I tried working on apt after the hang, I would get a message that the /var/lib/dpkg/lock file was being held by a process. How was I to find that process? Well, run the command

ps aux | grep dpkg

This listed the offending process. Then just run the command

kill xxx

where xxx is the process number.

Back to that emulator...

I then decided to just go ahead and try out Genymotion. I followed the install instructions given on the documentation site. I ran into issues where the needed Virtual Box stated that the kernel drivers were not installed. However, on checking and seeing that it appeared in the applications menu, I thought that that was enough for Genymotion. So I tried launching the application and ran into an error stating it needed the vboxdrv module. So I still neded to figure out how to get that module installed.

After searching around, I came across a forum post that recommended removing virtualbox

sudo apt-get remove virtualbox-\*

then downloading the package from VirtualBox.org and installing it manually.

sudo dpkg -i ./virtualbox-5.2_5.2.8-121009~Ubuntu~xenial_amd64.deb

I did that then ran into dependency issues. The installation process stated it needed libcurl4. I ran the command

sudo apt install libcurl4

Then repeated the virtualbox installation command. It installed just fine after that.

I then went back to Genymotion, executed it, and it started up without a problem.

The next stage was setting up a virtual device. The one I chose was a custom tablet running Android 8.0. Unfortunately, Whatsapp was not compatible with that device, so I had to select a different one. I chose the Samsung Galaxy 8. Same problem! What was going on? Back to searching for a solution. At some point, I decided to put back the custom tablet and tried installing Instagram. It installed without any fuss whatsoever! Just for giggles I tried installing Whatsapp again. Aye, it showed it was compatible and installed without any issue too! This I don't understand. The next time I have nothing to do I'll do a little digging to find out what the issue was. But for now, the apps are up and running. And I am a happy camper!

Not so fast! The following day when I tried to start Genymotion it failed! Reason? The needed Virtualbox modules did not get loaded! On looking at the /dev folder, those modules had gone missing! Back to troubleshooting mode...

In the applications tab, Virtualbox was shown as having been installed. And I could actually run it. However, it did issue a warning that the character device /dev/vboxdrv did not exist, stating that virtual machines would not get started without it.

I went back to the instructions from the Virtualbox forum and tried them again. In the process, I came across an error that stated insertion of the vboxdrv failed. Time to use another almost forgotten instruction to figure things out: dmesg. Running it, however, yielded no clues. On re-reading the error message from the build process, it showed that there was an error executing the vboxdrv.sh script. Looking through the script file, I couldn't pick out anything. And then, while thinking about it, I remembered reading in one of the numerous sites I found researching this issue yesterday that secure boot needed to be disabled for successful installation of Virtualbox. I thought that once installed, then I could re-enable it. Sure enough, after I disabled it again, everything worked just fine. But, I can't leave it unsecure, so I went looking to see if I could operate Virtualbox while still enabling secure boot.

The resource I found by Øyvind Stegard was super clear on resolving this issue. He explains in his blog that the vbox modules are not loaded since they are usigned. Ubuntu does not sign third party modules, suggesting that secure boot should be turned off to allow them. However, since that is not an option, he provides a way to sign them so that they are loaded just like any other modules with secure boot turned on. His instructions were very clear, and on testing after implementing them, I found (using lsmod | grep vbox) that the modules were indeed loaded. Genymotion, of course, started without any problems. I just need to remember to sign the modules each time the kernel is updated. Maybe I can find a way to automate this process, so that once an update is detected, the signing is activated. Hmm...

I know I had vented how yak shaving chased me from Windows. Yet, here I am, doing the same exact thing in Linux. That said, I think shaving yaks in Linux land is a much better and more exciting proposition than the same in Windows land. And I learn so much more in the process!

Two takeaways from all this...

Why was I willing to stay in the abusive Windows and Bluestacks relationships this long? Why? And for the Windows part, I knew long before of a better alternative! Bluestacks, I always had the inkling that there must be something better, but the trouble of researching, sorting out issues and installing an alternative seemed more of a problem than the abuse it was handing out to me. Ah, this masochism, never again!
I really should have stuck mucking around the kernel and other innards of things Linux way back when. How far would I be today had I continued along that path? It hit me the other day how at one point I was able to not only compile the linux kernel on my own after numerous customizations, but that I was also able to go into the actual source code and change things that didn't seem to work as expected for my machine. This, after successfully setting up Gentoo, then later, Arch. Today I can barely tell you the first thing about downloading the source code, let alone configuring it! Time to get back to that space, this time, permanently! And on that note, I think I'll revive that moribund laptop in storage just for this experimentation!

These chronicles are a way to record the steps I take in doing various things in the entire Linux ecosystem as a reminder to my future self on how to deal with problems I may have encountered in the past. Frequency of posts may be as many as several a day, or as few as some every few months.

References, links and notes:

There is indeed a book by that title. Do check it out https://www.goodreads.com/book/show/1871619.The_Joy_of_Linux

Ubuntu: https://www.ubuntu.com/

Cinelerra: http://cinelerra.org/

Blender: https://www.blender.org/

HandBrake: https://handbrake.fr/

GIMP: https://www.gimp.org/

RawTherapee: http://rawtherapee.com/

UFRaw: http://ufraw.sourceforge.net/

digiKam: https://www.digikam.org/

digiKam Recipes Book: https://www.digikam.org/recipes_book/

** Tutorial Purgatory https://medium.freecodecamp.org/how-to-escape-tutorial-purgatory-as-a-new-developer-or-at-any-time-in-your-career-e3a4b2384a40

Franz: https://meetfranz.com/

TechViola article: https://www.techviola.com/2018/04/android-emulator-ubuntu.html

Ask Ubuntu forum that explained how to get rid of the lock file in use problem: https://askubuntu.com/questions/15433/unable-to-lock-the-administration-directory-var-lib-dpkg-is-another-process

Genymotion: https://www.genymotion.com/

Fabian Lee's post on installing genymotion https://fabianlee.org/2017/09/23/ubuntu-installing-the-genymotion-android-emulator/

VirtualBox forum: https://forums.virtualbox.org/viewtopic.php?f=7&t=87335&p=422615#p422615

Øyvind Stegard's article on signing third party modules https://stegard.net/2016/10/virtualbox-secure-boot-ubuntu-fail/

Yak shaving: [MIT AI Lab, after 2000: orig. probably from a Ren & Stimpy episode.] Any seemingly pointless activity which is actually necessary to solve a problem which solves a problem which, several levels of recursion later, solves the real problem you're working on. http://www.catb.org/~esr/jargon/html/Y/yak-shaving.html