DEV Community: goatmale

Three things for 12/31

goatmale — Fri, 31 Dec 2021 17:19:23 +0000

Three things from this week:

/etc/.skel has files related to .bash_logout and .bashrc which are created for new users - I needed to remove the .bash_logout file for the issues with clearing screen that caused problems with the Gitlab runner and this seemed like the best way to do it.
Slack is actually an acronym that stands for Searchable Log of All Conversation and Knowledge useless, but fun :)
All about BuildKit in Docker - a simple environment variable DOCKER_BUILDKIT=1 enables performance improvements.

See you party people next week!!

Three things from today - 9/10

goatmale — Tue, 10 Sep 2019 19:48:55 +0000

9/10/2019

Tuesday.

A quote from Kahlil Gibran:

“Work is love made visible. And if you can't work with love, but only with distaste, it is better that you should leave your work and sit at the gate of the temple and take alms of the people who work with joy”

Three things simply from today:

1. "It's later than you think"

A seriously heartbreaking but moving story about loss. Helps keep perspective about what is important. Warning that this could make you tear up during a meeting.

2. Google's dataproc service is in alpha.

Cloud Dataproc is a managed Apache Spark and Apache Hadoop service that
lets you take advantage of open source data tools for batch processing, > querying, streaming, and machine learning.

This is a really interesting alternative to HDInsights or Databricks. I will need to toy with this later.

3. Senior Engineer checklist

A great list of standards to print out and aspire to if you're looking to grow your career.

See you party people Tomorrow!!

Three things from today - 9/9

goatmale — Mon, 09 Sep 2019 19:30:13 +0000

9/9/2019

Monday - last Monday before vacay :D

Hello everyone, I am just a week out from a week long Vegas vacation!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! - what can I say, i'm excited.

Hopefully your 9/9 is filled with nine nines of uptime.

OK - As the kids say (they don't) - let's get "learnt" up 🤢🤢

Three things that I learned & found handy today:

1. Handy reverse interview reference questions

This was posted on HN. I am a big fan of asking questions during interviews, this is a nice starter list of questions to ask. Don't be the guy that says "I don't have any questions". It makes you look like you don't care and it just generally doesn't lead to good discussion.
reverse-interview

One question that I don't see listed here that I saw posted on the Chicago tech community slack a while back was to ask the interviewer what brought you in. This kind of gives you an idea of what they think your strong suits are and you're able to reinforcece that to them.

2. Ubuntu and bluetooth issues

Some seriously helpful tips with getting bluetooth issues resolved.

Great info on how bluetooth works, and how to fix the darn headset when it's not working right.

3. Azure / AWS tips

Some quick Azure / AWS issues / tips I found today.

Azure - If you are unable to see the full list of VM sizes when trying to resize a VM in Azure, you may need to fully deallocate the VM. My thought is that this is probably limited to only the Hyper-V cluster that the machine is running on when the VM is running but deallocating allows it to be pushed to any host.
AWS - I found out today that you aws eks update-kubeconfig MERGES CONFIGS!!!

aws eks update-kubeconfig --name my-awesome-cluster

This is super handy if you use kubectx to manage multiple kubeconfigs as I do. Also pretty handy if you have a terminal like spacefish which displays the current context.

See you party people Tomorrow!!

Three things from today - 9/6

goatmale — Fri, 06 Sep 2019 21:56:01 +0000

9/6/2019

Friddddddddday

Friday - nuff said.

1. Undo git commit

...because I always end up looking it up -

git reset HEAD~

2. Tilix

I finally found a terminal manager that I like better than Terminiator!

3. OODA loop

Useful cycle for troubleshooting:

Observe
Orient
Decide
Act

See you party people Monday!

Three things from today - 9/5

goatmale — Thu, 05 Sep 2019 20:42:11 +0000

9/5/2019

Spark it up!

Today has been rough, but it's looking up. Three things incoming - ALL Spark related :)

1. cnvrg.io - spark-on-kubernetes webinar

I hope to finish this tonight it seems like a nice intro to Spark on Kubernetes and I'm digging it so far.

Cnvrg.io seems to be a botique data science platform similar to Databricks, I'll definitely have my eye on them.

2. Spark tuning for shuffles

Found this gem for spark tuning - a great outline on why shuffles occur and what you can do to tune them.

3. Spark exit status and exit codes.

Another great StackOverflow find regarding Spark exit status and exit codes.

See you party people tomorrow!

Three things from today - 9/4

goatmale — Wed, 04 Sep 2019 20:00:18 +0000

9/4/2019

Back to being busy all the time!

Today has flown by as i'm trying to keep up with everything!!!

I feel like i'm learning new things about how I work as I get older. One of the things i've realized recently is how the most important skill for anyone being overwhelmed is to break everything down into smaller chunks, focus on one priority chunk and work slower and more deliberately.

It feels counter intuitive but it's really the only thing that makes sense. Working slower, you can still take time to learn things and work smarter which ultimately pays off in the long run. :)

Anyway - three things!

1. Azure AKS known issues list

known k8s on azure issues and fixes

An Azure engineer who goes by the handle andyzhangx has a great resource of common issues when using AKS. Anyone interested in Azure Kubernetes Service or Azure Kubernetes Engine should keep this handy for when they run into a wall troubleshooting an issue.

2. Docker / R package install gotchas.

Working on updating a Docker image I noticed that some of the dependencies that we thought were in the image were not working. Our Rscript was super simple:

repourl <- "http://cran.rstudio.com/"

install.packages("randomForest", repos = repourl)

Turns out -

RScript doesn't care whatever happens in the script, the return code is always 0!
The randomForest package was missing the "gfortan" package (blast from the past)

I fixed the issue by installing the package but I also had to update our Rscript to tryCatch and quit with exit code 1 on any errors so that Docker would exit if any package installs failed:

repourl <- "http://cran.rstudio.com/"
tryCatch(
         {
                 install.packages("randomForest", repos = repourl)
         },
         warning=function(warning_message) {
         message(warning_message)
         quit("yes",1)
                },
         error=function(error_message) {
         message(error_message)
         quit("yes",1)
                }
         )

3. Darknet Diaries - Xbox Underground

Darknet Diaries - Xbox underground

I finished a podcast I enjoyed immensely this morning.

It's a two part story on a hacking group who hacked around with the Xbox 360.

It's a wild ride including hacking, drugs, friendship and video games. I highly recommend it.

Also on my reading list is the Wired article for the same story.

See you party people tomorrow!

Three things from today - 9/3

goatmale — Tue, 03 Sep 2019 21:39:54 +0000

9/3/2019

Happy Post-Labor day!

Hope everyone's Labor day was wonderful.

1. Overwrite entrypoint for Non-interactive Docker container

Monkeying around with a Docker image today I found I needed to run some commands on a non-interactive container.

Luckily I was able to find the exact command I needed to shell in and do my work.

docker run -it --entrypoint /bin/bash <image>

2. What the heck is R?

As I am finding myself being drug (dragged?) into the world of data science kicking and screaming i'm finding the world is wrought with acronyms, disparate and similar programming languages, and all sorts of other demons.

Today's demon is R.

R is a interpreted language (yay no compiling) which is used primary for statistical computing.

I was able to install the Deb for https://www.rstudio.com/ - which seems like the best IDE for the platform.

From there I was able to draw a pretty graph with some of the demos.

First, I pulled some data from here regarding the Bulls in '93

I saved it as a CSV and was able to load it into RStudio and show a pretty Pie chart with the commands below:

#Read data from CSV
data = read.csv("bulls.csv")
#Let's see the data with just Player and points.
data.frame(x=data$Player,y=data$PTS)
#Pretty piechart!
pie(data$PTS,labels=data$Player)

Finally, data science can tell us who the best Bulls player is!

3. Kubefwd

"Bulk port forwarding Kubernetes services for local development."

Putting a pin in this project - this seems to be super handy for developing locally on minikube.

It's benefit seems to be its simplicity.

Only one command and you can forward all of the service ports associated with the specified namespace.

Instead of messing around with host entries of forwarding ports, you can let this do it for you.

See you all tomorrow!

Three things from today - 8/30

goatmale — Fri, 30 Aug 2019 21:12:51 +0000

8/30/2019

Happy Friday!

I had a bit too much 🍷 yesterday so i'm a bit foggy, however that's not stopping me from learning today!

1. Spark S3 tweaking

Spark appears to use ParquetOutputCommitter - a function of Hadoop to write parquet files to S3.

Digging into an issue we've had writing Spark to S3 we came across a fix described here which involves setting a config value in hadoop:

Testing it - it appears to work.

I made a PR to set this going forward from the Hadoop / Hive side of things.

 hadoop_conf.set('spark.hadoop.mapreduce.fileoutputcommitter.algorithm.version', '2')

This can also be set in Spark with the following property:

'spark.hadoop.mapreduce.fileoutputcommitter.algorithm.version':2

2. Helm 3 beta 2 is released

The buzz around the office is that Helm 3 beta 2 has been released. Helm 3 is an important release for Helm as it removes the dependency on Tiller.
It is so important it seems to have warranted a 7 part blog series on their website

Also Microsoft has a great article describing why Helm 3 is important.

3. What is Apache Zeppelin? How is it different than Jupyter?

Apache Zeppelin and Jupyter are both interactive notebooks that you can use to do data science things like perform calculation, plot graphs, etc.

Jupyter notebooks run python in the background. Apache Zeppelin uses JVM underneath the hood.

As for features I enjoy, Jupyter is an offshoot of iPython, which I enjoy quite a bit for doing Python work.
Apache Zeppelin seems to be a little more robust for non-python languages, and also their demo is pretty sweet, being able to use their sweet Angular graph UI is pretty swell.

Check out these graphs!

4. Bonus thing!!

I have been promising myself i'm going to learn prometheus for too long. It's time to dig into the awesome-prometheus list....

I also created a new GKE cluster on my own Google Cloud account for testing.. Compared to Azure and AWS it's Kubernetes easy mode. More to come with monitoring and my new test cluster next week :)

Happy Labor day - seeya Tuesday!

Three things from today - 8/29

goatmale — Thu, 29 Aug 2019 18:50:18 +0000

8/29

Back at it again.

Hello everyone! Hope you're having a great Thursday. I know I am. I have the joy of sharing with you three things I learned or found interesting today.

1. What does Kubernetes kube-proxy actually do?

As i've been learning K8s i've been trying to break down each component into quick understandable parts. Today I did a quick dive into kube-proxy.

So what does kube-proxy actually do?

Kube-proxy exists as a pod on each node.
Kube-proxy manages the virtual IPs for services.
In most versions, by default, Kube-proxy uses iptables to do this.
- iptable rules interact with the Linux "netfilter" kernel hooks to make
- this happen quickly and without much overhead.
This allows you to use a virtual IP to route service traffic to various backend pods.
Kube-proxy acts as a watchdog watching the k8s api and if a service changes it updates the associated IPtables rules.

By viewing the logs of a kube-proxy pod you can see which mode you are using:

kubectl logs kube-proxy-qfqxk
W0804 15:49:33.304448       1 server.go:198] WARNING: all flags other than --config, --write-config-to, and --cleanup are deprecated. Please begin using a config file ASAP.
I0804 15:49:33.618663       1 server_others.go:148] Using iptables Proxier.

Neat!

2. Linux programming interface - New book i'm reading

Ok, i'll be honest, having a ton of books at your disposal with a service like Safari leaves you reading the first chapter and really not much else for a lot of books. I'm hoping to actually keep this up and learn a lot of the low level magic that makes Linux work.

3. SafeEyes 👀 - FOSS Linux Eye strain prevention tool

This tool has been really helping my eyes lately. Every 15 minutes it asks you to do things such as stare off into the distance for 15 seconds which helps not only your eyes, but also to take a quick break and empty your mind. I was worried about the context switching, but for me this doesn't seem to impact my productivity negatively, in fact I think it's actually helping.

Three things from today - 8/28

goatmale — Wed, 28 Aug 2019 20:50:48 +0000

8/28

Another day in the life.

I want to get into the habit of sharing with the community so i'm starting a week-daily post where I share three(ish) things i've learned or found interesting while I nurse my afternoon tea.

1. Kubernetes.Academy

Source: Coworker
Why I like it: This launched recently and seems like a great way to get started with Kubernetes. Between Octant and this, they seem to be killing it at VMWare.

2. Csysdig tool

Source: Csysdig blog post I stumbled on.
I have been using the standard Sysdig command line tool to capture traces for the other engineers to review however today I just discovered the interactive ncurses interface that you can use with csysdig.
You can easily use F2 to jump over to containers and then F6 to dig as you normally would, but now there are a ton of other things to play around with.

Also, pretty colors:

3. "Negative" subprocess error codes in Python.

Source: This Stackoverflow page I found while troubleshooting an issue.

Often times I troubleshoot exit codes as a means to determine why something failed. Today's issue was with a Python exit code of -11. Hidden in the comments it's written that these negative return codes correspond to the typical Linux signals. This is specifically when using the Python subprocess module, nevertheless handy to note these signals for troubleshooting.

Signal    Name    Description
SIGHUP    1    Hangup (POSIX)
SIGINT    2    Terminal interrupt (ANSI)
SIGQUIT    3    Terminal quit (POSIX)
SIGILL    4    Illegal instruction (ANSI)
SIGTRAP    5    Trace trap (POSIX)
SIGIOT    6    IOT Trap (4.2 BSD)
SIGBUS    7    BUS error (4.2 BSD)
SIGFPE    8    Floating point exception (ANSI)
SIGKILL    9    Kill(can't be caught or ignored) (POSIX)
SIGUSR1    10    User defined signal 1 (POSIX)
SIGSEGV    11    Invalid memory segment access (ANSI)
SIGUSR2    12    User defined signal 2 (POSIX)
SIGPIPE    13    Write on a pipe with no reader, Broken pipe (POSIX)
SIGALRM    14    Alarm clock (POSIX)
SIGTERM    15    Termination (ANSI)
SIGSTKFLT    16    Stack fault
SIGCHLD    17    Child process has stopped or exited, changed (POSIX)
SIGCONTv    18    Continue executing, if stopped (POSIX)
SIGSTOP    19    Stop executing(can't be caught or ignored) (POSIX)
SIGTSTP    20    Terminal stop signal (POSIX)
SIGTTIN    21    Background process trying to read, from TTY (POSIX)
SIGTTOU    22    Background process trying to write, to TTY (POSIX)
SIGURG    23    Urgent condition on socket (4.2 BSD)
SIGXCPU    24    CPU limit exceeded (4.2 BSD)
SIGXFSZ    25    File size limit exceeded (4.2 BSD)
SIGVTALRM    26    Virtual alarm clock (4.2 BSD)
SIGPROF    27    Profiling alarm clock (4.2 BSD)
SIGWINCH    28    Window size change (4.3 BSD, Sun)
SIGIO    29    I/O now possible (4.2 BSD)
SIGPWR    30    Power failure restart (System V)

4. Bonus! - Kubectl describe pod

This is kind of silly, but I figured out that I don't need to type the full pod name when using "describe pod" it seems to glob the name :D

Doesn't work.

kubectl get pod kubernetes-dashboard
Error from server (NotFound): pods "kubernetes-dashboard" not found

Works!

Name:               kubernetes-dashboard-7947fffdf5-f9vz5
Namespace:          kube-system

What's the best way to report a security hole to a small company?

goatmale — Fri, 26 Jul 2019 21:47:12 +0000

I was looking for some help on an issue I was facing and I accidentally stumbled upon a service from another company which was exposed to the public internet.

This service allowed exporting the raw configuration, API keys, usernames, passwords, urls, etc. D:

I tried to find a contact to reach out to, but I wasn't able to find anything on their site. The contact on their site is only for sales and press, of which I want neither.

How would you handle this situation?