DEV Community: Michael Butak

Deciding Between On-Prem & The Cloud

Michael Butak — Tue, 19 Dec 2023 23:37:33 +0000

For organizations with existing on-prem workloads, deciding what to move to the cloud can be tricky. This decision tree will help bring clarity to the question:

Using Little's Law for Cloud Infrastructure Planning

Michael Butak — Thu, 24 Aug 2023 16:51:38 +0000

I thought I'd share a bit about Little's Law and how it can be useful for infrastructure planning purposes.

Little's Law is a mathematical theorem from the science of queuing theory. It's very simple but very powerful for modeling everything from airport security lines to bus routes. Before showing an example of how to use it for cloud resource planning, let me explain the law itself:

Little's Law states that the (average) number of units in a process at a given time (L) is equal to the (average) arrival rate (λ) times the average amount of time each unit spends in the process (W).

Thus:
L = λ * W

Using an example from AWS Lambda, it may be used to estimate how many concurrent lambda functions might be required to support a certain process if we can estimate the request rate and the average duration of each invocation. Since the duration of each lambda invocation is a metric available in the AWS console, you need only estimate the number of requests per second to derive the number of concurrent lambda functions you should expect, on average:

request rate (λ): 100 (requests per second)
duration per invocation (W): 1 (second)
number of units in process at a time (L): L = λ * W

so:
100 requests/sec x 1 sec = 100 requests in progress at any given time

Thus we can determine that a process with these assumptions will require an average of 100 concurrent invocations of the lambda function at any given time. This process will not likely ever hit AWS's default soft limit of 1000 concurrent instances.

If the process expected 600 requests per second and each request had an average duration of 2 seconds, then we would be worried about the limit of 1000 concurrent instances, because we would expect 600 x 2 = 1200 requests to be in-process at any given time.

Thus, even though lambda is a service that scales automatically, there is a concurrency limit, and we ought to proactively check whether there is a risk of hitting it. Little's Law is a quick way to check.

Note that the formula can be manipulated to solve for other variables too:

L = λ * W (Calculate the number of units in a process at a given time)
W = L/ λ (Calculate the amount of time each unit spends in the process)
λ = L/ W (Calculate the arrival rate of new units into the process)

Handling JSON in Python

Michael Butak — Thu, 04 Nov 2021 18:07:52 +0000

You might wonder, "json.load, json.loads, json.dump, json.dumps oh my!" Which do I use for which situation?

json.load
<- you want to read a json file and access its contents as a python object. Example:

You have a file named 'some-json.json' which is a dict with properties "A" and "B". In python you could access it this way:

import json
some_dict = json.load(open("some-json.json", "r"))

print(f"Property 'A' is {some_dict['A']}")
print(f"Property 'B' is {some_dict['B']}")

json.loads
<- you want to convert a json string into a python object. Example:

You have a string like:

some-json-string = ‘{“A”: “BOB”, ”B”: “JOHN”, “C”: “SAM”}’

And you wish to access its properties. You could do it with json.loads (where the 's' stands for string):

import json
some-json-string = ‘{“A”: “BOB”, ”B”: “JOHN”, “C”: “SAM”}’
some_dict = json.loads(some-json-string)
print(f"Property 'A' is {some_dict['A']}")
print(f"Property 'B' is {some_dict['B']}")

json.dumps
Next we have json.dumps, which allows us to make a json string (hence the 's' in 'dumps') out of a python object. An example:

import json
some_dict = {
    'A': 1,
    'B': 2,
}

serialized-json-ready-to-send = json.dumps(some_dict)

json.dump
Last we have json.dump, which is used to write python to a json file. Example:

import json

# content to write to json file:
some_dict = {
    'A': 1,
    'B': 2,
}

with open("output.json", "w") as outfile:
    json.dump(some_dict, outfile)

How to export .cer and .key from .pfx file

Michael Butak — Thu, 01 Apr 2021 21:40:18 +0000

So you are ready to add a signed certificate and private key to your web application. Here's one approach to making that happen.

Note, I'm using a Windows 10 workstation.

For the purpose of this post, we assume you already have a .pfx file from your certificate authority. (.pfx is a binary file. There are other file formats such as .pem (which is base64 encoded). If you have a .pem file you can convert it to .pfx and then follow these steps.)

Save the .pfx file on your computer. In my examples below, the pfx file is saved at C:/Users/usernameGoesHere/.ssh
Next you will need to extract the .key and .cer files from the .pfx:
Ensure you have openssl installed.
In this example the openssl.exe executable is installed at /bin/openssl
From the dir on your workstation where you have the pfx file from your CA (in my example named my-site.com.pfx), run the following command:

echo QUIT | /bin/openssl.exe pkcs12 -in my-site.com.pfx -nocerts -out server.cer.key -nodes

NOTE: If you are using something other than the git bash command line emulator, you might not need the echo QUIT | part, and you can replace /bin/openssl.exe with just openssl.)

NOTE2: We have included the -nodes flag so that the key is not encrypted with an export key.

Note that you now have a server.cer.key file in your directory.
Next run this command to extract the .key:
echo QUIT | /bin/openssl.exe pkcs12 -in my-site.com.pfx -out server.cer -nokeys -clcerts

NOTE: same as above, (if you aren’t using git bash emulator, you might no need the first bit of the command)

Promises in JavaScript

Michael Butak — Tue, 30 Mar 2021 15:01:15 +0000

What is a Promise?

A promise in JavaScript is a mechanism for handling asynchronous code (like a disk read or a network request). What does that mean? A synchronous method returns its resultant value immediately (immediately meaning as soon as it’s done evaluating the expressions it contains). An asynchronous method doesn’t return its value immediately. It has to wait for something else to complete (https request, etc.) before it has an “answer”. Because JavaScript is single-threaded, and it can only process instructions in a single sequence, one-command-at-a-time, this presents a problem when one of those commands is asynchronous. Any tasks subsequent to it requiring as an input the result of our asynchronous operation are liable to evaluate their expressions before that value is available and our code will fail.

Promises present a solution to this problem. They make asynchronous code behave more like synchronous code. The way they do this is to give the asynchronous code something else to immediately return as soon as it is evaluated while waiting for the resultant value. What is this something else it returns instead? A promise. The promise stands as a proxy to the final value until the final value becomes available. And to this promise object we may attach callback functions which do not get invoked before the completion of the current run of the event loop. This means, the final value will be available at the time when those callbacks are invoked. (This is critical for understanding promises.)

A promise contains state that is not accessible to any function but the async operation that created it. The state is either pending, fulfilled, or rejected. Pending is the initial state of the promise. It has neither been fulfilled nor rejected. Fulfilled means the asynchronous operation has completed successfully. Rejected means the operation failed. We can instruct the promise to call different callbacks based on whether the promise settles in a fulfilled or rejected state.

These callbacks are added to the runtime message queue by the promises’ .then and .catch methods. (See diagram here) I have found this understanding helpful in the practical implementation of promises.

A site for practical examples of how to use promises: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Guide/Using_promises

A site with a fascinating introduction to combining promises with event emitters: https://www.jpwilliams.dev/using-eventemitters-to-resolve-promises-from-afar-in-nodejs

Feel free to improve upon this explanation by leaving a comment.

How to export/import custom content from ManageIQ environments

Michael Butak — Mon, 21 Dec 2020 19:14:52 +0000

This post is a how-to specific to ManageIQ development.

Whereas custom automate code can be exported from the ManageIQ UI, other custom content in ManageIQ (like Service Dialogs, custom buttons, etc.) must be exported via the command line. Below are the steps for exporting content from one environment and importing it to another. This does require some basic knowledge of linux administration:

Steps:

Create an exports folder at /tmp/exports on source server and an imports folder at /tmp/imports on destination server.
From within the /var/www/miq/vmdb directory on the source server use the rake commands from the image above to export one category of items into that folder. E.g.,: rake evm:export:service_dialogs -- --directory /tmp/exports/
Use scp to move the contents of /tmp/exports on the source server to the /tm/imports dir on the destination server.
From within the /var/www/miq/vmdb directory on destination server use rake commands from the image above to import one category of items at a time. E.g., rake evm:import:service_dialogs -- --source /tmp/imports/
OPTIONAL:
a. Delete /tmp/imports on destination server
b. Delete /tmp/exports on source server
Repeat from step 2 through all the various categories of items to export.

Here's a whole article: https://access.redhat.com/documentation/en-us/red_hat_cloudforms/4.7/html/scripting_actions_in_cloudforms/migrating_custom_buttons

Adding SCM Credentials To ManageIQ

Michael Butak — Mon, 21 Dec 2020 16:23:09 +0000

Credentials are a critical component to application integrations. Debugging credential errors can be maddening. In this post I’m sharing my learnings on how to add Source Control Management (SCM) credentials to ManageIQ (to be used, in this case, for importing Ansible Playbooks for use in Embedded Ansible). Special thanks to @fryguy, @NickLaMuro and @pemcg for their support in figuring this all out.

My instructions are GitLab specific, but GitHub should be very similar:

It is advised that we use ssh (not https) for maximum security.

Steps to generate credentials and pull in a repo from GitLab:

Generate a pair of ssh keys (on your local machine or on a server). There are many guides online for how to do this using ssh-keygen. Unless you opt to name them something else, it’ll create a public key file called id_rsa.pub and a private key file called id_rsa. In your command line you can use cat file_name to view, select, and copy the contents. (You’ll need this to paste the keys in GitLab and ManageIQ.) Put the public key in gitLab. (Got to your profile->Settings-> SSH keys.) Note that: a. The name of the key you save is not part of the authentication process so you can name it whatever you want. b. Extra spaces or carriage returns in the key (happens at the beginning/end of the text usually) can mess it up. c. The SHA256 fingerprint is displayed after you save it. You can always come back and view it later if you need to debug your keys, but for now just note that it’s available in GitLab when you open a key.
Create a new credential object in ManageIQ: a. In MIQ, go to Automate–>Ansible–>Credentials, under Configuration choose ‘Add A New Credential’ b. Name the credential c. For Type choose Scm (Source Control Management, i.e., Git) d. For username choose “git”. e. Leave password blank f. If you used a passphrase when generating your ssh keys provide that in “Private Key Passphrase”. Otherwise leave it blank. g. Paste your private key in the field named Private Key. This would be the file contents of the id_rsa file generated when you created your keys. h. Click Add
Now you can add a repository in ManageIQ. a. Click on Automation --> Ansible–> Repositories. Under Configuration choose Add new Repository. b. Name it c. Under URL provide the SSH url from your project in GitLab. Note that it should begin with git@your-teams-gitlab.net…. and end with .git. d. Under SCM Credentials select the credential object you just created in step 3. e. Optionally choose the branch. I recommend you choose master. f. Click Add
Now you can see your playbooks (if there are any in your GitLab project) under Automation–>Ansible–>Playbooks

How To Debug your Credentials if It Fails to Sync the Repository from GitLab
Basically you need to confirm that the public key you put in the credential object in ManageIQ has a corresponding public key in GitLab. The other thing that can go wrong is the passphrase used to create the keys in the first place. If you forgot your passphrase or don’t remember if you used one, you’ll need to start over and get that part right. But assuming that’s not the problem, you need to figure out whether it’s your private key in ManageIQ or public key in GitLab that’s missing/messed up.

Check your key fingerprints on your machine where you created the keys: a. Both the private and public key have the same fingerprint. It can be in one of several formats (SHA256 or MD5). I’ll use SHA256 in these exampls. b. On your local machine, cd into the dir where your id_rsa and id_rsa.pub keys are. c. Run ssh-keygen -lf id_rsa to print the SHA256 fingerprint of the private key. d. Optionally run ssh-keygen -lf id_rsa.pub to print the SHA256 fingerprint of the public key. These two will match.
View your SSH key fingerprints in GitLab a. Go to profile–>settings–>ssh keys and inspect each key to see if any has a SHA256 fingerprint matching what you found on your machine where you created the keys. If not, then you haven’t properly saved your id_rsa.pub key to GitLab. b. Note that extra spaces or carriage returns in the key can mess it up, as mentioned previously. c. If you find a matching fingerprint, proceed to check the keys in ManageIQ.
View your known hosts on the ManageIQ server a. Whenever you add a private key, it adds the finger print of the key as a known host to a key file found at ~/.ssh/known_hosts b. Run ssh-keygen -l -f known_hosts and it will print a list of fingerprints on the list. c. Check whether the fingerprint from step 2 and 3 is also found in this list. If not, then there was a problem with how you set up your credential object with the private key. Delete the object in ManageIQ and do it over following the steps above. d. If the fingerprint does not appear in the list, you can also inspect your credential object to pull the fingerprint from it directly: i. From /var/www/miq/vmdb enter the rails console by typing rails console. ii. Once in, run this command (updating the name of the credential to whatever you named it:) GitRepository.find_by(:name => "name_of_credential").authentication.auth_key This should print the contents of your private key. To extract the fingerprint, we need to do a few more steps: iii. Also in the rails console, run: File.write("/tmp/fookey", GitRepository.find_by(:name => "name_of_credential").authentication.auth_key) This will generate a key-like file called fookey in your tmp folder. iv. Run chmod 600 /tmp/fookey so that so that, (U)ser / owner can read, can write and can’t execute. v. Get out of the ruby console, and cd /tmp vi. Run ssh-keygen -y -f fookey3 > fookey.pub to create a public key from the private key. vii. Run ssh-keygen -lf fookey(same as from step 2 above) to print the fingerprint. If this doesn’t match your fingerprint in GitLab, you won’t be able to authenticate.

How To create an HTTPS connection with GitLab
As mentioned above, it is recommend to use an SSH connection, but it is possible and valid to use https.

Create a new credential object in ManageIQ: a. In MIQ, go to Automate–>Ansible–>Credentials, under Configuration choose ‘Add A New Credential’ b. Name the credential c. For Type choose Scm (Source Control Management, i.e., Git) d. For username and password, provide a username and password that have access to your repository in GitLab. e. Leave the following fields blank: “Private Key Passphrase”, “Private Key” f. Click Add
Now you can add a repository in ManageIQ. a. Click on Automation --> Ansible–> Repositories. Under Configuration choose Add new Repository. b. Name it c. Under URL provide the https url from your project in GitLab. Note that it should begin with https://…. and end with .git. d. Under SCM Credentials select the credential object you just created in step 1. e. Optionally choose the branch. I recommend you choose master. f. Click Add
Now you can see your playbooks (if there are any in your GitLab project) under Automation–>Ansible–>Playbooks

I suspect the same process would apply more or less for adding credential objects to AWX/Ansible Tower. Others can comment on whether or not that is the case.

Spend Less Time Being STUCK

Michael Butak — Mon, 21 Dec 2020 16:18:21 +0000

Spend less time being STUCK: We software developers get "stuck" at times. I personally don't enjoy the experience. As I reflected on this problem, I realized that there must be strategies I could use that would reduce the amount of time I spend being stuck and thereby increase my productivity. After doing some research on this topic (links at the bottom of the page), I found the following solutions:

Prevent getting stuck in the first place:
- Analyze requests to break them down into tasks that can be assigned to time boxes. Time-boxing provides a signal (task takes more time than expected) as to when you are probably stuck. The sooner you realize you’re stuck, the sooner you can take action to get unstuck.
- Diagram the problem: Do you understand the logical flow of what you are trying to do? Can you diagram it out?
- Pseudo-code: Did you plan out the algorithm in detail in plain English before diving into the code?
- Avoid self-sabotaging habits that increase chances of getting and staying stuck:
  - multi-tasking (lack of focus)
  - randomly choosing work (lack of prioritizing)
  - randomly attacking work (lack of planning hours of the day)
  - pet projects (lack of focusing on customer request)
Recognize that getting stuck is a learning and growth opportunity. You are about to learn something and be a better developer as a result.
Read the error carefully. Very often the answer is in the error message.
Check for the obvious errors:
- Typos
- Wrong types (string instead of “string”)
- “=” instead of “==”
- Missing brackets
Verbalize the problem: (aka the Rubber Duck method). Put into words (out loud) in layman’s terms: what am I trying to do? How am I trying to do it? What is happening instead? What evidence do I have that I should expect the intended result?
Break the problem down: sometimes we get overwhelmed by the many moving pieces. If we break the problem down into pieces, we can efficiently fix/test/build each part in sequence until we have a solution.
Knock out the easy parts: If there’s a portion of the problem that you’re good at and know how to do, go get that part done now. The success can help relieve frustration and clear your mind.
Refocus on the customer’s original request: Ask myself,
- “Am I down a bunny trail… or focused on the original request?”
- "Have I inadvertently added to the scope of the request… or am I focused on the original request?”
- "Have I gotten caught up on one particular solution… or am I focused on the intent behind the original request?”
Simplify: Ask myself,
- “Is there a simpler way to accomplish this?”
- “Am I trying to boil the ocean?” (I.e., trying to make the end-all-be-all solution to solve everything instead of focusing on a practical, simple solution)
Name the mystery: Oftentimes when we get stuck it is because we are making use of a tool/technology/language that is new to us. What “black box of ignorance” might you be running into? What do you need to research up on in order to fill in the black box with knowledge and understanding? (E.g., “what is an SSH key anyway?”)
The usual strategies: (print statements, log reading, googling error messages, Stack overflow posts, asking colleagues, taking a break.)

And most importantly: Pray for understanding: “If any of you lacks wisdom, let him ask God, who gives generously to all without reproach, and it will be given him.” (James 1:5) Don’t believe me? Try it.

Some helpful links I found:
https://www.youtube.com/watch?v=h01U6uDhNk4
https://hackernoon.com/5-steps-in-programming-to-keep-you-from-getting-stuck-g52f233gb
https://codewithoutrules.com/2016/12/08/how-not-to-get-stuck/
https://www.freecodecamp.org/news/how-to-get-unstuck/