DEV Community: Joe Zack

Making Money with Code

Joe Zack — Mon, 10 May 2021 03:18:41 +0000

Congratulations! If you know how to code, then you are a highly skilled worker. You can use those skills to make money, either by trading your time (active income) or creating automated income streams to make you money "while you sleep".

What follows are the notes for the episode, which you can find wherever podcasts are found - just search for "Coding Blocks"!

Show Me the Money

Active Income

Active income is income earned by exchanging time for money. This typically includes salary and hourly employment, as well as contracting.
Some types of active income blur the lines.
Way to find active income can include job sites like Stack Overflow Jobs, Indeed, Upwork, etc.
- Government grants and jobs are out there as well.
Active income is typically has some ceiling, such as your time.

Passive Income

Passive income is income earned on an investment, any kind of investment, such as stock markets, affiliate networks, content sales for things like books, music, courses, etc.
The work you do for the passive income can blur lines, especially when that work is promotion.
Passive income is generally not tied to your time.

Passive Income Options

Create a SaaS app to keep people coming back. Don’t let the term SaaS scare you off. This can be something smaller like a regex validator.
Affiliate links are a great example of passive income because you need to invest the time once to create the link.
Ads and sponsors: typically, the more targeted the audience is for the ad, the more the ad is worth.
Donations via services like Ko-fi, Patreon, and PayPal.
Apps, plugins, website templates/themes
Create content, such as books, courses, videos, etc. Self-publishing can have a bigger reward and offer more freedom, but doesn’t come with the built-in audience and marketing team that a publisher can offer.
Arbitrage between markets.
Grow an audience, be it on YouTube, Twitch, podcasting, blogging, etc.

Things to Consider

What’s the up-front effort and/or investment?
How much maintenance can you afford?
How much will it cost you?
Who gets hurt if you choose to quit?
What can you realistically keep up with?
What are the legal and tax liabilities?

Resources We Like

Apply for Grants To Fund Open Source Work (changeset.nyc)
Government grants and loans (usa.gov, grants.gov)
35 Passive Income Ideas for Developers [All Types] (beginnerspassiveincome.com)
8 Side Income Ideas For Programmers (That Actually Work) (afternerd.com)
Podcasts
- The Smart Passive Income Podcast with Pat Flynn (smartpassiveincome.com)
- Entrepreneurs On Fire (eofire.com)
- How I Built This with Guy Raz (npr.org)
Who Moved My Cheese (Amazon)
The Lean Startup: How Today’s Entrepreneurs Use Continuous Innovation to Create Radically Successful Businesses (Amazon)
These top Patreon creators earn more than $200,000 a year (blog.patreon.com)
Ko-fi – Make an Income Doing What You Love (ko-fi.com)
PayPal – Make your Donate button (paypal.com)
How Long Does It Take To Create An Online Course? (onlinecoursehow.com)
Udemy – Planning your online course (udemy.com)
Am I Procrastinating? (amiprocrastinating.com)
Google Ads Help: Use Keyword Planner (support.google.com)

Tip of the Week

Google developer documentation style guide: Word list (developers.google.com)
In Windows Terminal, use CTRL+SHIFT+W to close a tab or the window.
The GitHub CLI manual (cli.github.com)
- Use gh pr create --fill to create a pull request using your last commit message as the title and body of the PR.
- We’ve discussed the GitHub CLI in episode 142 and episode 155.
How to get a dependency tree for an artifact? (Stack Overflow)
xltrail – Version control for Excel workbooks (xltrail.com)
Spring Initializr (start.spring.io)
- You can leverage the same thing in IntelliJ with Spring.

Getting Started with K9s

Joe Zack — Sat, 24 Apr 2021 03:09:00 +0000

In this post I’ll tell you a bit about K9s, which is an open source, command line interface for Kubernetes that makes me feel like a super hero. I’ll tell you what K9s is, why I like it so much, and what the trade-offs are. I hope that to convince you to give it a shot, and that you’ll love it as much as I do.

Check out our quick walkthrough on YouTube

What is K9s?

Open Source Cross Platform CLI for Kubernetes
Intermittently polls cluster resources
Quickly navigate contexts, namespaces and resources
Quickly Filtering, sorting, and copying
Steep, but short learning curve

If you have worked with Kubernetes for very long, then you know how often you have to “kubectl”. It’s a lot, probably a “babillion” times per day. There are clever short-cuts you can take, but ultimately the tool is a thin veneer over the Kubernetes API. There are benefits to working so closely with the API itself, but ultimately it’s a lot of work to do fairly routine tasks.

Think about how Often you “get pods”, “logs”, or “port-forward”. What about those times you need to look up a secret or edit a custom resource you can’t quite remember the name of? Have you ever made your changes only to later realize you altered the wrong cluster or namespace? Oops.

K9s solves these problems by focusing on the user experience, crafting an interface to the Kubernetes API that is convenient for humans. K9s presents you with a resource oriented view that offers filtering, sorting, and context-sensitive actions. It does this via a command-line interface, with a heavy emphasis on short-cuts. The command-line nature and short-cuts are a large part of what makes K9s so great, but they are also the source of it’s learning curve.

11/10 This is a good dog

Why do I like K9s?

Speed:

Refreshing in the background
- No more looking up pod names by label or autocomplete (if you’re on bash)
Swapping namespaces and Contexts
Multiple instances (no changing of default)
Saving, Filtering logs
Sorting
Easy secrets

Visibility:

Always know your…
- Namespace
- Context
- Cluster
Doesn’t change your default context/namespace

Bonus points

Pulses
Popeye
Accidentally learning new things
Easy to install on remote servers

What can I say? I love how K9s makes me feel. It polls for resources in the background, which makes it fast for me to zip around between screens. This also makes it really easy to filter and sort tabular information. Filtering is done via the “/” key, it supports wildcards, labels and information that isn’t immediately available on the screen (More akin to grepping a “kubectl describe” than a “kubectl get”, if that makes sense). Filtering also works on all views, including logs and yaml.

There are context specific actions available as well. For example you can hit “shift-f” to port-forward on a page with services, deployments or pods. You hit “l” for logs, “s” to shell in, “ctrl-d” to delete and so forth.

If you work in a multi-cluster and/or with multiple namespaces then you will be glad to know that the cluster, context and namespace are visible in the top area. Even better, swapping the context (:context or :ctx) or the namespace (:namespace or :ns) does *not* change your global context. It’s like passing –context and –namespace along with every command. This makes it easy to work with multiple clusters at once, even with multiple instances of K9s.

Everything I described here is fast and simple but it takes sometime to build that muscle memory. There are only a few things you need to remember. I posted a video at the top of this article, but here is the short of it. Once you’ve internalized those keystrokes, the rest will fall into place and you’ll be zooming around like a superhero in no time.

Just remember this:

ctrl-a (See all the things, including custom resources!)
: navigates (contexts, namespaces, tools, and resources)
? for help
/ filters (works on just about anything)

One more quick thing I want to mention? Ever heard of pulses, popeye, or x-ray? These are all tools that I learned of because I saw them with ctrl-a. I’ve also bumped into other resources (custom or otherwise) that I hadn’t heard of before. K9s made it easy to discover these tools and resources, and I feel like I’m constantly bumping into something new.

Lonely sad puppy. Ecotional scene. 11/10

What are the trade-offs

kubectl is ubiquitous
You can’t script k9s
Batch operations aren’t fantastic either
Large clusters can be slow (Drop that poll time down!)
“Freemium” (There are plans for k9s to offer a paid, premium version in the future)

If you can’t tell by now, I love K9s. I’ve shown it to many people at this point, and I want to mention the common arguments I hear against using it. For the most part they really just boil down to benefits of sticking with kubectl. Of course, there is nothing stopping you from using kubectl at any point but there is something to be said for sticking with one tool…especially when that tool is completely ubiquitous.

Conclusion

As you can see, I’m rather fervent about it because k9s makes me feel like I have super powers. I feel fast, safe, and like I’m getting stronger everyday. I had to spend some time up front getting used to it, but now I feel like I’m flying. I never thought I’d be making such grand statements about a command line interfaceface tool…but here we here. Now hit the links and give it a shot! Feel free to join the slack, or reach out to me if you have any questions!

Links:

The Scariest Bug (I Haven’t Written Yet)

Joe Zack — Mon, 06 Jul 2020 03:39:52 +0000

Okay, so I promised to sing a song if Coding Blocks got enough podcast reviews and, thanks to you , we made it kinda, and I sang a song kinda!

Okay, so I actually dusted off a song I had written a while back for a Halloween gag I never went through with but..dang, I cannot sing. Rather than subject you to my warbling, I hired an amazing singer on fiverr. Trust me, it’s better this way.

Anyway, here it is – a short song, with video, about the inevitability of making programming mistakes. Enjoy!

Getting started with Apache Beam and Kotlin

Joe Zack — Wed, 08 Apr 2020 18:48:35 +0000

Are you looking for a simple getting started guide for working with Kotlin and Apache Beam? Well congratulations, you are in the right place. I’ve been doing some live coding on the subject, and I made a teeny tiny project to help me get started on new Beam projects quicker.

There is an official guide for getting started, so you should probably start there. However, that guide begins with a fully functional project and I prefer to build a minimum viable product from scratch that is easy to run in an IDE. So if you are like me, and prefer to type your own code then keep reading!

Note: If you don’t like typing either, you can just clone the repo and you are ready to go

Wait, what is Beam?

Beam is an open-source and unified model for interacting with streaming and batch data. Have you heard of projects or methodologies like Spark, Flink, MapReduce or batch processing? If not, then this post will not be useful for you. If you have, though, and you haven't heard of Beam - then please let me introduce you!

Beam is a project that lets you choose a supported language (currently Java, Python, and Go), and write code can run in either batch or streaming mode, and can run in a variety of engines (Spark, Flink, Apex, Dataflow, etc). That means you get to focus on your business logic, and let Beam worry about the implementation. Check out the compatibility matrices for more information.

Is this the future of batch and streaming data?
Maybe! Time will tell but I think it's worth investigating if these areas interest you.

Create the project

First up, “File” > “New Project” and choose “Maven”. Check the “Create from archetype” checkbox and select the org.jetbrains.kotlin:kotlin-archetype-jvm , because Kotlin is awesome.

Choose the Kotlin JVM archetype, but you can use the default settings for everything else.

Add the minimal dependencies

I’m aiming for a bare-bones implementation here, so I’m just going to add the dependency for the direct runner directly to my project for now. Later we can break this out to a “profile” so you can optionally compile in the bits for the execution engines of your choice.

<!-- Add these to your dependencies in pom.xml -->
<dependency>
  <groupId>org.apache.beam</groupId>
  <artifactId>beam-sdks-java-core</artifactId>
  <version>2.19.0</version>
</dependency>

<!-- This dependency is for the local runner. -->
<!-- Later we can move this one out to a profile -->
<dependency>
  <groupId>org.apache.beam</groupId>
  <artifactId>beam-runners-direct-java</artifactId>
  <version>2.19.0</version>
  <scope>runtime</scope>
</dependency>

Create a simple pipeline

Now let’s setup a simple pipeline that is essentially a stripped down version of the official example.

I’ll paste the code below, but here is the outline of what is happening:

Create a pipeline with default options
Add a step to read from pom.xml
Add a step to split the text into text characters
Add a step to count the characters
Add a step to format the output to a collection of strings
Add a step to write the output to files, prefixed with “counts:
Execute the pipeline

fun main(args: Array<String>) {
    val p = Pipeline.create()
    p
            .apply<PCollection<String>>(TextIO.read().from("pom.xml"))
            .apply(
                    FlatMapElements.into(TypeDescriptors.strings())
                            .via(ProcessFunction<String, List<String>> { input -> input.toList().map { it.toString() } })
            )
            .apply(Count.perElement<String>())
            .apply(
                    MapElements.into(TypeDescriptors.strings())
                            .via(ProcessFunction { input -> "${input.key} : ${input.value}" })
            )
            .apply(TextIO.write().to("counts"))

    p.run()
}

Now you can run the code, and you’ll end up with several files in your project directory of the format counts-0000*!

(note: if your main method does not show up as runnable, you may need to right-click pom.xml and “Maven” > “Reimport”)

What is next?

I plan to use this project as a jumping off point for experimentation, but I also have some small plans for this project on it’s own.

Here is what I’m looking at doing to this repo next:

Adding tests
Adding profiles for multiple runners
Reading from a streaming source of data

Let me know if you’re interested in hearing more about Beam and I will be happy to write more about it.

Check out our YouTube channel for more information and live streaming with Beam!

The Best Dev Audiobooks 🔊

Joe Zack — Sun, 08 Mar 2020 03:34:41 +0000

I’m a big fan of audio learning, and of course there are tons of great, free, dev podcasts but did you know that there are quite a few great audiobooks for devs as well?

I've been a subscriber to Audible.com (and yes, these are affiliate links below) for a long time, but they have only recently started adding the types of programming books that I liked to read. I've kept track of the Audiobooks that I’ve picked up that intersect with books you frequently see on "best of " lists. I’m proud to say I’m 6.5/7 through the list already and I've got the next one already on deck!

The Pragmatic Programmer

Frequently cited as the most popular programmer book of all time, this book needs little introduction however I recently noticed that the newly updated 20th anniversary edition is available on Audible and it’s next on my list even though I already read the print version.

The DevOps Handbook

This is the book I’m currently “reading”. It starts off a bit slow if you’re already familiar with the notion of DevOps and don’t need convincing. The rest has been pure gold though, and I find myself reaching for my note-taking app over and over again.

The Phoenix Project

The Phoenix Project is fiction, but it’s an inspiring story that weaves in many ideals of change management and DevOps from the DevOps handbook.

The Unicorn Project

This book is from the perspective of a different person on a different team, but roughly at the same time as The Phoenix Project. It was released 6 years later though, and it’s been updated a bit to reflect more recent technologies. The book is standalone though, so no ned to read the other first.

The Manager’s Path

Okay, okay, I know what you’re thinking but this book is really helpful for anybody who has ever been in a dev lead or management position or wants to be in one someday *or* has ever been managed.

The Lean Startup

This is the popular book that people frequently regarding Minimum Viable Products, “in”-trapreneurs and iteratively delivering value.

Masters of DOOM

Ok, so this is not directly biz or dev related but it was still an interesting book for anybody who is interested in games, the games industry or “crunch time”. I can’t believe how many innovations came out of Id Software!

Conclusion

Well, that's my list. I'd love to hear your recommendations!

My 2020 (Dev|Re)solutions

Joe Zack — Thu, 02 Jan 2020 03:08:26 +0000

New Year’s is my favorite holiday because I find the notion of periodic renewal invigorating. Let that old lasagna be forgot, I want to move forward! Overall, I was pretty happy with how my technical skills progressed last year, but I think I can do better. Much better.

This year I’m done trying to micromanage my future with limited information. I want to keep things simple by setting a high level focus and prioritizing habits over static goals.

Focus: (Near) Real-Time Analytics and User Experiences

The software that people are interacting with gets better and better every year: better looking, more intuitive, and more responsive. That’s great news for the internet, and people, and everything I guess. That also means that people are coming to expect more and more from my software too. Yikes.

Apps like Uber, Tinder, and Slack offer great user experiences and the technologies underneath the UI iceburg are really cool too.

Since this is a high level objective, I’m not going to specify any more detail than that. I want to keep my plan lean and adaptable so I can hop on to any needs or opportunities that 2020 swings my way. I also don’t want to feel obligated to stick to an out-of-date plan.

Here’s a short list of technologies that are often associated with the (near) real-time types of experiences that I’m talking about:

Pub/Sub queues like Kafka or AWS Kinesis
GraphQL subscriptions
Apollo Client
Web Sockets
Reactive Programming

Uber has a fantastic user experience, and my customers are coming to expect more real-time capabilities in the software that I write.

Creating Habits: Deliberate scheduling time for tasks

I tend to spend a lot of time researching things for work, Coding Blocks, or other various side-projects* so finding tech time has never been a problem for me. I spend way too much time on this stuff, but I enjoy it and it’s a passion of mine. Because of that, I’ve always entertained a bit of chaos in my extracurricular activities – letting myself work on whatever I felt inspired to work on. Any why not, this is my “free” time! The problem with this approach is that it’s all too easy over spend time on things that are easy or urgent…continuously bumping important/non-urgent and more mundane tasks to the back of the line.

I aim to mitigate this habit somewhat this year, by creating a new one. I want to spend 3 hours a week (in 1 hour chunks) working on deliberate, planned tasks. I’ll decide ahead of time what the task is, then I’ll set a timer, and do it.

Any tech time above and beyond that, including time for blogging, presentations, making videos, or…whatever I feel like is fine too – but I’m going to give this dedicated time a whirl to see if I do a better job of staying on top of important, but not urgent, tasks.

Conclusion

I’m really looking forward to this year, and I hope that my plan of setting a single high-level focus and scheduling deliberate time for it works out. And if it doesn’t? Well, then I’ll stop and try something else! It’s not much of a “resolution” but I think it might “just work” for me.

Pic: https://elements.envato.com/traffic-in-hong-kong-PZ9DFG3

* Family, friends, and a hobbies too – but this is a tech focused blog, yo!

Streaming process output to a browser, with SignalR and C#

Joe Zack — Tue, 17 Dec 2019 05:01:00 +0000

I’m happy to announce that I’m taking part in the The Third Annual C# Advent this year along side many great articlesand podcasts. If you have even a passing interest in C# then you should go check it out and follow all of the authors on social media!

Okay, enough with the niceties! See, when it comes to holidays I am more of a Grinch than a Griswold so I’m going to show you how to do something in C# that is cool, powerful, and … dangerous! I’m going to show you how, from a website, to invoke processes in a command shell and stream the results back to the browser via SignalR.

And before we get any further I should note, this blog post will not be a good introduction to SignalR, that documentation already exists. This article is intended for developers who have web development experience with C#, and are at least somewhat familiar with the concepts of SignalR, and web sockets.

Finally…

Warning! We are literally about to create a remote code execution injection vulnerability, as it allows any user or process with access to the website to run arbitrary commands. However, there are times when you may want to do something similar, very carefully and in a limited fashion. Check out Azure Cloud Shell and Coder.com for examples of web apps that let you interact with shells…in a browser.

Ok so what exactly are we doing?

Creating an ASP.NET web app that lets users run arbitrary cmd.exe commands on the webserver shell
Streaming STDOUT and STDERROR output to the browser via SignalR
Exploring ideas for how we might evolve this project into something really, really cool

Note: The full source code is available on Github in case you hate typing: https://github.com/codingblocks/streaming-stdout-with-signalr

This is what we’ll end up with, a simple and dangerous way to execute server-side commands in a web browser. Awesome.

Let’s write some code!

I’ve tried to keep things simple so that you can focus on the meat of the message. The downside is, that you may run into some dragons if you stray off the beaten path. Stray anyway! Please feel free to drop a comment with any questions or comments about things you run into.

Step 1: Create a new ASP.NET Core Web Application (tested with v3.0.101)

Assuming you have dotnet installed, the following command will set you up with a basic razor pages website, in a new directory named “streaming” :

dotnet new webapp -o streaming
cd streaming

Step 2: Add a dependency for SiginalR

SignalR is a Microsoft project that makes pub/sub interactions easy for a variety of different client and server technologies. In this example we’ll be using SignalR to stream output from our process, and we’ll also be using a SignalR JavaScript library on the front-end to subscribe to this output and display it on the website. We’ll only be using a small portion of what SignalR offers in this example, but hopefully it’s enough to whet your appetite.

This command will install the server-side bits, and we’ll have a client-side js file to add in the front-end when we get there

dotnet add package Microsoft.AspNetCore.SignalR.Client

Step 3: Server-Side Changes

Now we’re ready for some code. Let’s create a class for managing our processes, ProcessHub.cs. Make it inherit from the SignalR “Hub” class so that it can access the relevant communication channels. I’m trying to keep the noise down in this post, so I’m only going to focus on the interesting bits, but the full file is on Github,

{
  // I'm keeping a static copy of the process around, so that we don't end up spawning a new one every time the user asks for one. This also makes the app single threaded, which is most appropriate for many use cases anyway!
  TerminateProcessIfExists();

  var process = new Process()
  {
    // Note: You can swap this for the particular shell you want to run, but I'm keeping this simple so sticking to windows with "cmd.exe"
    StartInfo = new ProcessStartInfo("cmd.exe")
    {
      // The process does not allow for redirection by default, so we need to instruct the process to explicitly allow it
      RedirectStandardOutput = true,
      RedirectStandardInput = true,
      RedirectStandardError = true,
      // This is also required for redirecting output, "Shell" here refers to an actual graphical shell - which isn't going to work for us here
      UseShellExecute = false,
    }
  };

  // These events will trigger our custom outputHandler, which we'll see in the next code section
  process.OutputDataReceived += new DataReceivedEventHandler(
    (sendingProcess, outLine) => outputHandler(outLine.Data)
  );
  process.ErrorDataReceived += new DataReceivedEventHandler(
    (sendingProcess, outLine) => outputHandler(outLine.Data)
  );

  process.Start();
  // We have to explicitly start our redirection in addition to starting the process
  process.BeginOutputReadLine();
  process.BeginErrorReadLine();

  // Notice that we started the redirection BEFORE we executed the command
  process.StandardInput.Write(command + NEWLINE);

  // The process will return async by default, so we have to instruct the runtime to hang on to the process so we keep the redirections alive
  process.WaitForExit();
  _process = process;
}

The method below is what SignalR actually calls, note that we pass in an Action that will send our output back to subscribing clients whenever it’s called.

// The command argument is an arbitrary shell command that we will execute on the server:
public void Stream(string command)
{
  // This calls our private method with the command passed from the client (this is super dangerous, never do this) as well as an Action that will publish output (o) text to any clients that are subscribed to "OutputReceived". Note: this proof-of-concept only has one subscriber,
  Stream(command, o =>
  {
    Clients.All.SendAsync("OutputRecieved", o);
  });
}

We’ll also need to make two minor changes to Startup.cs in order to hook things up: (Reminder: full file on Github)

// Add this line to the ConfigureServices method
services.AddSignalR();

// Add these lines to the bottom of the Configure method to set up the url for the client to call:
app.UseSignalR(routes =>
{
  routes.MapHub&lt;CommunicationHub>("/stream");
});

Step 4: Client-Side Changes

Here’s a handy article for adding SignalR to the front-end in Visual Studio. or you can just use a cdn to keep things simple. The user interface is really minimal here, just a text input and a button. The output of the commands will just stream to an empty div below.

The JS shown below is mainly glue code, but it shows how we listen for the button press event, and create / start a new connection.

// Invoke the "Stream" method, with the command
document.getElementById("executeButton").addEventListener("click", () => {
  var command = document.getElementById("commandText").value;
  var connection = new   signalR.HubConnectionBuilder().withUrl("/stream").build();
  var output = document.getElementById("output");
  output.innerHTML = '';

  connection
    .start()
    .then(() => {
      connection.on("OutputRecieved", message => {
        var p = document.createElement("p");
        p.textContent = message;
        output.appendChild(p);
      });
      connection.invoke("Stream", command);
    }).catch(err => alert(`An error occurred: ${err}`));
});

A good example command to run is “ping codingblocks.net” since it shows you what it looks like as data comes in piecemeal. Other cool examples to try “git status”, “dir”, or “ssh”.

Here is a sample of how things are looking now

Pretty neat right? At this point, the project is just a toy but I bet you can think of some cool stuff to do with it. Like...

Okay, now what?

This project is a simple proof of concept showing how you can call processes in C#, and stream the output to a website. It’s not too useful by itself, but there are a myriad of cool ways you can evolve this project to make something really cool! Here are a couple ideas to get your brain juices flowing:

Make a dashboard website for custom scripts on your computer
Make a UI for git
Make a UI for checking out Docker logs and exec-ing (I did this!)
Make a full on browser based terminal using xterm, including STDIN and a shared session
Add basic functionality to current websites for things like ping, grep, nmap, or telnet
Fix all the minor bugs in this project (like, what happens when you run two commands without refreshing!)

Hopefully you got something out of this post. Make sure to check out the other Advent of C# posts.

Here’s a screenshot of a simple Docker admin page I setup to view streaming logs, very similar to this project.

In Defense of Big O

Joe Zack — Sun, 06 Oct 2019 18:10:36 +0000

One of the few things that developers can agree on is that the interview process is terribly broken. There are ideas about how to fix it but the industry has yet to come to a consensus on a better way forward.

Many developers are particularly aggravated by the way some organizations insist on “academic” interview problems that test developers skill with algorithms and (Big O) algorithmic analysis. The argument is that these types of problems are often irrelevant to the job, rewarding antiquated knowledge over practical skill, and are biased in favor of developers with more time and resources to spend on preparation.

Now, before we continue let me get this out of the way:

I agree with those arguments!

https://www.xkcd.com/2146/

As a developer with social media accounts I am also inundated with jokes, memes, and tweets that essentially boil down to “Big O is dumb” and I worry that reductionist platitudes are unfairly tainting how new developers view important tools and techniques.

Sometimes it’s really important

Maybe your dream job doesn’t require a lot of data and compute, but that doesn’t mean it’s not important for others. Algorithms, data structures, and Big O can certainly come in handy here, but so does system design, scalability, and having a good grasp on the numbers. I don’t know how to rank the importance of these subjects but I’ll wager that the developers working on things like databases, search engines, operating systems, and graphical rendering (and many other fields) are critically dependent on having a shared and unambiguous language for understanding how programs scale in terms of space and time complexity.

If you are interested in these types of subjects, then you are going to need to study up on the underlying principles. I agree that the developers who face these types of problems generally have more time than an interview takes to optimally solve them but it is important to show your interviewers that you share (or are capable of) the same basis for precise communication.

Examples:

Also, algorithms are awesome!

Having a good understanding of algorithms and data structures is like a super power. I’ve seen (and probably written) lots of …uh…interesting…. solutions for dealing with dependency management when a simple “top sort” would have done the trick faster and more correctly. Being able to recognize and classify a problem can go a long way towards solving it. Additionally, perusing algorithms can lead to sparks of inspiration, letting you solve problems that you didn’t even realize you had.

Here’s a list of known algorithms to get your motor running:

https://en.wikipedia.org/wiki/List_of_algorithms

Summary

Interviewing is difficult with many organizations cargo culting the FANG interview process. Despite that, the study of algorithms, data structures, and algorithmic analysis is still really useful for many programmers, and it’s also kinda awesome.

Tell ’em Barack!

PS: Looking for some resources to get started? Check out these great articles on dev.to!

PPS: Like this kind of topic? Check out the podcast in your app of choice!

Why Avro?

Joe Zack — Thu, 12 Sep 2019 01:53:04 +0000

Apache Avro is an open source data serialization system that lets you send information. It is frequently associated with “big data” and distributed systems because it has some distinct advantages over the competition.

The primary advantages are listed below, but read on for more information:

Messages are highly efficient
Strong schema support
- Schema versioning
- Dynamic schema support
- Different, but compatible, Reader and Writer versions allowed
- Union types
Object container files can include schema along with encoded records

Why not human readable formats?

There are many different ways of serializing and deserializing data, and they each have their own advantages. JSON is probably the most common format that developers deal with today, but XML is still king in some tech stacks. The prime benefit of these formats is that it easy for humans to read, making development and debugging much easier.

However, the same things that make formats like JSON and XML human readable (like labels for each element or attribute) are the same things that make it grossly inefficient.

Don’t believe me? Here’s a link to the Pokemon pokedex in json format. The labels (“id”, “name”, “Attack”, “Sp. Attack”) make up more data than the data does!

Yes, You can compress json and xml, but it’s expensive and your data is no longer human readable…so why not choose a better format at that point?

Pros :

Human readable / writable
Optional schema
Widely supported
Great for light tasks

Cons :

Computationally inefficient

What about Protocol Buffers/Thrift?

Formats like Protocol Buffers or Thrift are not human readable, but they are dramatically more efficient. The idea is that you can create a schema that defines the shape of the data, and you can share that schema with anybody who needs to decode your data.

This allows us to dramatically reduce the amount of space per message. The schema also id based, which offers a limited kind of schema evolution where you can add or remove (as long as they aren’t required) new fields without breaking anything.

Pros:

Highly efficient
Most major languages have 3p library support for it
Limited support for schema evolution
Required schemas can serve as documentation

Cons:

Schemas must be shared between serializers and deserializers
Can’t add or remove required fields
Schemas are not directly versioned
Complicated for light tasks

Why Avro?

I listed all the reasons above, and I’ll list them again below, but ultimately it boils down to having awesome schema support.

Having a stand-alone, versioned schema allows Avro to keep the minimum amount of information in it’s messages – making them highly compact. The schema support also allows for different readers and writers to negotiate on which schemas they support, which bakes a lot of flexibility in at the lowest level.

There are some additional niceties that go along with this too, like making it easier to support dynamic schemas rather than compiling them in with your code and having to figure out how to share the schema across projects. Sure, you could build this on top of another format…and also build that support into all of the tools you support, but why do that when you could use a popular, agreed upon standard?

Pros :

Messages are highly efficient
Strong schema support
- Schema versioning
- Dynamic schema support
- Different, but compatible, Reader and Writer versions allowed
- Union types
Object container files can include schema along with encoded records

Cons:

Support is not as wide as other formats
Complicated for light tasks

Read a much better comparison here: https://martin.kleppmann.com/2012/12/05/schema-evolution-in-avro-protocol-buffers-thrift.html

The author of that post also wrote this awesome book!

, and if you're interested in this post then I bet you will love it.

Analyzing Dev.to Git History with Elasticsearch

Joe Zack — Tue, 07 May 2019 04:15:20 +0000

Importing Git History into Elasticsearch is not very much fun thanks to the lack of git formatting options, but it's pretty cool once you get it working!

The commits that I imported has 22 data points, including a one-to-many list of changed files. I imported the git history from dev.to to see how things looked.

Now I can run simple queries like this to see things like how many commits were made by each contributor:

That's nice and all, but I'd prefer something visual so I put together a simple dashboard in Kibana.

This is a screenshot of the charts I put together. I can apply filters and searches at the top of the dashboard and it will affect all the charts.

I can (for example) look at commits by Ben Halpern in the last 90 days that involve "Algolia".

Check out the video to see a bit more about the visualizations:

Pretty cool right?

Conclusion

Eventually I'd like to take some time to clean-up the import process so it's easier to run, but for now I'm going to take some time exploring a couple of my own projects to see if I can dig anything interesting up.

The Pragmatic Programmer: Investing in Your Knowledge Portfolio

Joe Zack — Mon, 29 Apr 2019 00:00:24 +0000

Coding Blocks just began a new series of podcasts about my favorite all-time programmer-y book: The Pragmatic Programmer. Talking about it helped me focus my thoughts on the subjects, and I hope that you get as much out of the conversation as I did.

Listen to the show and let me know if you can figure out how to tell if you are making stone soup or just boiling frogs.

Here is a snippet of the show notes:

A Pragmatic Philosophy

Note: 20th anniversary edition going into beta May 8th

Written by programmers, not designers of some framework or language, the book presents patterns for developing software.

This book is packed full of greatness even in the foreword:

There are no perfect tools, methodologies, or solutions. You must pick the best bits for each particular situation.
- Don’t fall in love with any particular technology or tool.
- Use your experience to help pick the right solutions for any particular situation.
Pragmatic programmers get the work done and do it well.

This book is for those who want to be more productive and effective developers.

Traits of a Pragmatic Programmer

How would you rate yourself on these traits?

Early adopter / fast adapter
- Instinctive and passionate about trying out new technologies.
- Confident and grasp new things quickly.
Inquisitive
- You ask a ton of questions, you want to know as much as you can about what’s being done.
Critical thinker
- Don’t accept whatever is being said or claimed. Use your experience to think through the problem.
Realistic
- You understand when something is complex and what that means for timelines.
Jack of all trades
- You keep your knowledge broad even if your practice is very focused / narrow.

Tips

Care about your craft.
Think about your work.
- Always evaluate what you’re doing and why.

Just Good Advice

Embrace individuality and craftsmanship, even on large teams.
Always be working to sharpen and hone your skills.

The Cat Ate My Source Code

“The greatest of all weaknesses is the fear of appearing weak.” – J. B. Bossuet

Adopt an attitude, style, and philosophy of approaching problems and solutions where you’re always thinking about the big picture.
Take responsibility for your actions.
- Commit to something being done right even though you don’t have control over all aspects of what’s being done.
- This doesn’t mean you have to fall on the sword. In “impossible” situations, it’s up to your discretion to not accept responsibility for what is happening.
- Accepting responsibility is being held accountable.
- If you do make a mistake or there is a shortcoming, be honest and offer options.
  - Vendor didn’t come though? You should have had a contingency plan. Can’t blame things on them.
Being responsible doesn’t mean being perfect. It means working to prevent mistakes, and then working to fix them when things go wrong.
Be willing to change, but be cautious at the same time.
Know how good your software has to be. Define what is good enough?

Tip

Offer options, not excuses.
- “Rubber Duck” it. If what you have to say sounds lame, then you should probably save your breath and find some possible solutions.
- Admit when you need help or more resources.

Software Entropy

Entropy refers to the amount of “disorder” in a system.
Also known as “software rot” .
- Despite best plans and best people working on a project, it can still suffer from this.

Tip

Don’t live with broken windows.
- Don’t leave poor decisions or bad code in your code.
- If you don’t have time to fully fix it, patch it somehow.
- Neglect accelerates rot, rot begets rot.
- Don’t let the “the rest of the code sucks” mentality sink in, otherwise it will decline faster than you could imagine.
- Keeping the codebase clean effectively makes others not want to mess it up.
- If you don’t have time to do things properly, maybe consider “boarding it up”, i.e. comment the code out or display a warning.

Stone Soup and Boiled Frogs

Tip

Be a catalyst for change
- Talking about change often goes nowhere.
- People want to plan meetings.
- Then management approvals.
- Barriers to entry pile up.
- This is known as “start-up fatigue”.
- Start making strides in the direction you see the change needs to happen, show people, and they’ll begin to rally around you.
- People find it easier to join an ongoing success.

Tip

Remember the big picture, don’t be the frog in boiling water.
- Constantly review what’s happening around you otherwise you’ll lose sight of what’s happening and things can start to deteriorate without your noticing.

Good-Enough Software

Users should be able to participate in determining if something is good enough.
The scope and quality of the system you create should be part of the project specifications.
Knowing when to stop iterating over the same code is important.
- Code can never be perfect so leave well enough alone.
- Don’t over-embellish or over-refine an otherwise perfectly good program.

Tip

Make quality a requirements issue.
- Letting your users experience the software now may lead to better overall software in the future, even better than the path of perfecting what is there now.
- Great software today is better than perfect software tomorrow.
- How can we set standards around quality? Test coverage, code reviews, static code analysis, etc. maybe?

Your Knowledge Portfolio

“An investment in knowledge always pays the best interest.” – Benjamin Franklin

Your knowledge portfolio is all the knowledge you’ve gained programming both in the technical as well as domain specific bits you’ve picked up over time.
Your knowledge and experience are your most important professional assets … but are expiring assets.
Consider investing in your knowledge portfolio like you would make financial investments:
- Invest regularly, as a habit.
- Diversify.
- Manage risk.
- Buy low, sell high.
- Review and Rebalance.

Tip

Invest regularly in your knowledge portfolio
- Learn at least one new language a year.
- Read a technical book each quarter.
- Read nontechnical books, too.
- Take classes.
- Participate in user groups.
- Experiment with different environments.
- Stay current by subscribing to trade magazines, journals, newsletters, or similar.
- Join and participate in online communities and read white papers for technologies of interest.
- Be on the lookout for opportunities. If you can’t answer a question … don’t let it go!

Tip

Critically analyze what you read and hear.
- Try to think critically about what you read and hear, don’t take it for truth.
- Beware of zealots, or any time somebody can confidently tell you “the way”.

Communicate!

Ideas are worthless unless you can communicate it
Meetings, tickets, wikis, and code are all forms of communication.
Understand what you want to say.
- Jot down your high level goals.
- Have a plan for getting it across.
Know your audience.
- The WISDOM acrostic:
- W hat do you want them to learn?
- What is their I nterest?
- How S ophisticated are they?
- How much D etail do they want?
- Whom do you want to O wn the info?
- How can you M otivate them to listen to you?
Choose your moment wisely and make what you’re saying relevant in time.
Communicate with your audience the way they want to be communicated with.
Make your message look good.
- Presentation is as important as the content itself.
- Which is easier to read? One full page of text? Or two pages of text broken apart into sections with section headers?
Get people involved in your documents early and often.
- Involving your audience builds relationships and creates better documents.
Be a listener. If you don’t listen to others, they won’t listen to you.
Get back to people. People hate when you don’t reply, even if it’s just an “I’ll get back to you later.”

Tip

It’s Both What You Say and the Way You Say It.

Resources We Like

The Pragmatic Programmer by Andrew Hunt, David Thomas (Amazon)
The Pragmatic Bookshelf (pragprog.com)
Programmers’ Build Errors: A Case Study (at Google) (static.googleuserconent.com)
Broken windows theory (Wikipedia)
How ‘Broken Windows’ Helped Shape Tensions Between Police And Communities (www.wnyc.org)

Tip of the Week

Git the source code for your favorite old games. (GitHub)
Learn about all of your favorite design patterns using The Catalog of Design Patterns. (refactoring.guru)
Learn VS Code tricks you didn’t even know about. VS Code can do that?! (vscodecandothat.com)
Sharpen your Azure skills with these self paced labs provided by Microsoft:
- Acquire the cloud skills you need, at your own pace (handsonlabs.microsoft.com)
- Welcome to Self-paced Labs (www.microsoft.com)

Quickstart Guide: Elastic Stack for Devs

Joe Zack — Sun, 28 Apr 2019 20:03:37 +0000

Curious about the Elastic Stack? This quick guide will walk you through setting it up and importing articles from some of my favorite dev.to authors so you can play around!

Elasticsearch is an open-source, scalable, full-text search and analytics engine. It is used for a variety of purposes, from Full Text Search, to E-Commerce, to Real-Time Analytics. It is frequently associated with big organizations and big data, but Elastic did a great job with their default configurations so it's easy for smaller projects as well.

In this guide, we're going to spin up a basic Elastic Stack (formerly known as the "ELK Stack") that brings together a few different open-source services that were designed to work together. Here is a high level description of those services:

Elasticsearch is a persistence engine and API layer.
Logstash is a plugin-based tool for importing data
Kibana is an administration GUI for exploration and management

I also recorded a video if you prefer watching over reading:

Prerequisites

You need Docker for this guide, check here for installation instructions.

If you are on Windows, then I recommend using the Linux containers (default) and also sharing a drive so that Docker can persist your Elastic data to disk.

#1: Compose the architecture

The first step is to create a docker-compose.yml file that describes how your services fit together. I'm aiming for speed over depth in this guide so I have already created this for you.

Simply clone the repository and fire up the Elastic Stack:

git clone https://github.com/codingblocks/simplified-elastic-stack.git cd simplified-elastic-stack docker-compose up -d

Give Elasticsearch a few seconds to catch it's breath after it starts up and then you can verify it's status by hitting this url a browser: http://localhost:9200

Note: The logstash container will shutdown quickly because it doesn't yet have anything to do yet, so lets fix that!

#2: Import data

Now we'll use Logstash to import data. The repo you cloned above already has a custom Dockerfile.Logstash file, so lets add an input plugin that can import RSS feeds. All you have to do is add the second line to Dockerfile.Logstash so that it looks like this:

FROM docker.elastic.co/logstash/logstash-oss:7.0.0 RUN bin/logstash-plugin install logstash-input-rss

Now let's add a couple input configurations. Each "rss" block represents one RSS feed that will be imported into Elasticsearch every 3600 seconds. Replace the config/logstash.conf file contents with the following lines and Logstash will take care of the rest.

You can see the inputs are some of my favorite blogs, configured to poll once an hour. The output sets up a basic index called "blogs" that will hold the data.

Update config/logstash.conf:

input {
  rss {
    url => "https://dev.to/feed/davefollett"
    interval => 3600
  }

  rss {
    url => "https://dev.to/feed/dance2die"
    interval => 3600
  }

  rss {
    url => "https://dev.to/feed/dotnetcoreblog"
    interval => 3600
  }
  rss {
    url => "https://dev.to/feed/kritner"
    interval => 3600
  }

  rss {
    url => "https://dev.to/feed/molly_struve"
    interval => 3600
  }

  rss {
    url => "https://dev.to/feed/rionmonster"
    interval => 3600
  }

  rss {
    url => "https://dev.to/feed/TheInfraDev"
    interval => 3600
  }

  rss {
    url => "https://dev.to/feed/thejoezack"
    interval => 3600
  }
}

output {
  elasticsearch {
    action => "index"
    index => "blogs"
    hosts => "elasticsearch:9200"
    document_id => "%{[link]}"
  }
}

That's it, Logstash will take care of everything else. Next time we restart our environment, Logstash will start polling and importing the feed data.
Stop, re-build, and restart your environment:

docker-compose down docker-compose build docker-compose up -d

Give Elasticsearch a minute to breathe after docker-compose is running, and try hitting this url in the browser to see that you have data: http://localhost:9200/blogs/_search

#3: Have fun!

Everything is setup now, so it's time for you to do a bit of exploring.
If you are new to the Elastic Stack, I recommend getting acquainted with Kibana first. It's already running on your computer: http://localhost:5601
Head over to the "dev tools" and give a few of these queries a shot so you can get a taste of what Elastic has to offer.

Sample queries

Simple filter, Get the top 5 posts about JavaScript in the last year

GET /blogs/_search?q=JavaScript&size=5
  "query": {
    "bool": {
      "must": [
        {
          "range": {
            "published": {
              "gte" : "now-1y/y",
            }
          }
        }
      ]
    }
  }
}

Simple aggregate, posts by date

GET /blogs/_search?size=0
{
  "aggs":{
    "posts by date":{
      "date_histogram":{
        "field":"published",
        "interval":"year"
      }
    }
  }
}

Combination aggregate/filters: Top 10 results for Elasticsearch, with results and counts by author

GET /blogs/_search
{
  "query": {
    "bool": {
      "must": [
        {
          "match": {
            "message": "Elasticsearch"
          }
        }
      ]
    }
  },
  "aggs": {
    "author": {
      "terms": {
        "field": "author.keyword"
      }
    }
  }
}

Here are a couple suggestions for what to do now that you've got Elastic Stack up and running:

Build a simple website that lets you browse and search for your favorite blogs.
Explore the Search and Aggregations APIs* Create visualizations with Kibana

Troubleshooting: (Huge thanks to Dave Follett for the help!)

Running on linux, and the Elasticsearch container keeps crashing? Your vm.max_map_count might be too low. Check this out for an explanation / fix: https://www.elastic.co/guide/en/elasticsearch/reference/current/vm-max-map-count.html
Having trouble with containers? "docker logs" and "docker inspect" are your friends. You can run a command like this in order to see where all the elasticsearch files are on disk: "docker inspect $(docker ps -f name=elasticsearch -aq)"

Photo by Nine Köpfer on Unsplash