DEV Community: Marc Katz

Some basic bash tips

Marc Katz — Thu, 31 Aug 2023 02:45:20 +0000

When using a Unix terminal, executing commands can be complicated and time consuming to do manually. However, with the power of bash scripting they can all be automated and simplified. Here are a few basic tips you can use to spice up your command line experience:

Making a bash script

The first thing to do is to create a script file. This is a file with the suffix .sh. These files can be run from the command line with the command bash <filename.sh>. To the start of the file, add #!/bin/bash to tell the terminal how to run it.
In this file, you can do anything else you can in the command line, such as ls (list files in the directory), echo (print to the terminal), cd (change directory), and cat (write to file).

Variables and input

Like in other languages, you can create variables in a bash script. It's extremely simple, just <variable name> = <value>. If you want to access the variable later on, you can do so with $<variable name>. For example,

read user_input
echo $user_input

will simply print whatever the user typed into the command line back out at them.
For input, there are two main waits you can get it. One way is with read, which takes in input typed by the user in the command line. The other is with arguments, which can be passed when running the script initially. To access the arguments, use $<n>, where n is the nth argument passed.

Basic logic

The next step to make these scripts more programmable is with logic. The first main piece of logic is with if/elif/else statements. The main syntax is:

if [[<condition>]]; then
    <code>
elif [[<another condition>]]; then
    <code>
else
    <code>
fi

Don't forget to add fi to the end to end the if statement! You can also add as my elif statements as you need.
For the conditions, some basic logic operators include -a <filename> (returns true if the file exists), -s <filename> (returns true if the file exists and isn't empty), and -z <string> (returns true if the string is empty, -n is the opposite). For and, or, and not, you can use &&/-a, ||/-o, and ! respectively.

Loops

Lastly, we can add loops to the script. For a basic while loop, write

while [[<condition>]]; do
   <code>
done

For a for loops, do

for <var> in {<start>..<end>} do
   <code>
done

This will execute <code> <end>-<start> times, with the value stored in <var>

Finding Differences Two Ways: Bash and Diff

Marc Katz — Wed, 09 Aug 2023 14:36:18 +0000

For many projects, it is important to be able to figure out the changes that were made between two files or objects. This can be if you're recording edits, making incremental changes, or working collaboratively. In this post, I will describe how to do this through the Bash shell and with the difflib python library.

Bash diff Function

In the bash shell, there is a function diff that can directly compare two files (or directories). The basic format is simple, diff <file1> <file1>. This will output a string called a "diff" or "patch" describing what changes to make to turn file1 into file2. (Sidenote: if the files are identical, there will be no output, but if the files are not text files, it will output 0 if they are identical and 1 if they are not).

Options
There are many useful options to add to the diff command depending on what you need. Some include:

--ignore-space-change: ignore changes in whitespace
--ignore-blank-lines: ignore lines that are blank
--ignore-case: ignore changes in case
--minimal: "Try hard to find a smaller set of changes" - from the Man page
--recursive: when diffing directories, search through their sub-directories recursively
--side-by-side: output the changes in two columns, makes it much more human-readable.
--suppress-common-lines: don't output lines that are identical

Reading the output
Let's use the following example: You have two text files, a.txt and b.txt.
a.txt is:

This is line 1
This is line 2
This is line 3
This is line 4

b.txt is:

This is a new line
This is line 1
This is line 2 modified
This is line 3

If we run the following command: diff --minimal a.txt b.txt, we get the output:

0a1
> This is a new line
2c3
< This is line 2
---
> This is line 2 modified
4d4
< This is line 4

There are three sections of this output.

0a1
> This is a new line

The a in the first line means that you have to add the following lines. In this case, 0a1 means you add line 1 from b.txt to line 0 (the start of the file) to a.txt.

2c3
< This is line 2
---
> This is line 2 modified

The c means that this is a change. Here, line 2 in a.txt is replaced by line 3 in b.txt.

4d4
< This is line 4

The d means that there is a delete. Here, line 4 in a.txt is deleted.

difflib Python Library

However, if you want to do this in your code, you can't just use the bash shell. In Python, one way to do this is to use the difflib library. difflib is part of the Python Standard Library, so you don't need to install anything, just add import difflib to your file. There are many commands in this library, but the two we will go over is unified_diff and Differ.compare. We will be using these commands to compare two strings:
a:

This is line 1
This is line 2
This is line 3
This is line 4

and b:

This is a new line
This is bine 1
This is 2a
This is line 3

Note that these strings end with a newline, this is important. If you want to compare two string that don't end in a newline, you will have to add it to use these functions.

unified_diff
unified_diff is the simpler of the two commands, but does not give as much detail as the other one. The format is simple: difflib.unified_diff(list1, list2). The function takes in two list of strings, each ending in a new line. You can easily turn your string into a list like that by running <string>.splintlines(keepends=True). This function will output a "generator function", so you will have to turn it into a list, or immediately iterate through it. For this example, we will use the following code:

diffs = difflib.unified_diff(a.splitlines(keepends=True),b.splitlines(keepends=True))
for line in diffs:
    print(line,end='')

(The end='' is needed because each line of the output already ends in a newline)

Reading the output
For the above example, the output will look like:

--- 
+++
@@ -1,5 +1,5 @@
-This is line 1
-This is line 2
+This is a new line
+This is bine 1
+This is 2a
 This is line 3
-This is line 4

To read this, the first part of describes the lines of the string we are looking at. Here, we are looking at lines 1 to 5 in both strings. Lines starting with - will be deleted from the first string, and lines starting with + will be added from the second string. As you can see, this is very compact, but not very detailed, as it only works with adding and deleting whole lines

Differ.compare
Differ.compare will give much more detailed results than unified_diff. Like unified_diff, you have to use splitlines first, since it takes lists of strings. You also have to create a Differ object, since it is a function of that class. For this example, we will use the following code:

d = difflib.Differ()
results = d.compare(a.splitlines(keepends=True),b.splitlines(keepends=True))
for line in results:
    print(line,end='')

Reading the output
For the above example, the output will look like:

+ This is a new line
- This is line 1
?         ^
+ This is bine 1
?         ^
- This is line 2
?         -----
+ This is 2a
?          +
  This is line 3
- This is line 4

Like, unified_diff, lines starting with + are removed from the first string, and lines starting with - are removed from the second string. However, we now have these ? lines. These lines further describe the changes within each lines. If there is a ^ in that line, the character above the ^ in the line above will be replaced by another character above a ^ in the line below. So in the case above, the l in This is line 1 is replaced by a b. If there are -s, that means those characters are deleted from the line. Here, line is removed from This is line 2. If there are +s, that means the characters are added to the line. Here, an a is added to the end of This is line 2.
As you can see, this is much more complicated than unified_diff, but now you can see the exact changes you need to make to each line and where, instead of just completely removing and adding whole lines.

Sources:
https://ss64.com/bash/diff.html
https://docs.python.org/3/library/difflib.html#differ-objects

The Basics of Animation in CSS: Rotating and Scaling

Marc Katz — Wed, 26 Jul 2023 15:33:41 +0000

Animations in websites can be done in many ways. From coding it in vanilla Javascript, to the countless libraries you can download, to Flash (rip), you can pick and choose what you use to fit your needs. But one basic way to do so is in CSS. This is useful since they are natively supported by all the major browsers and don't require any extra downloads. In this post, we will show how to rotate and scale in multiple dimensions.

Setting the Stage

note: this will be done in SCSS
First, we need to set up the objects we want to animate. We will create 3 axes, and a circle at the center, all as divs. We'll wraps them in a single parent "scene" div. This will be all the HTML we'll need for this. So far, our code is:

<div class='scene'>
  <div class="axis x"></div>
  <div class="axis y"></div>
  <div class="axis z"></div>
  <div class='circle'></div>
</div>

However, if you look at this in the browser, you'll see nothing. So now, we need to add some basic properties to our divs.
For our body, let's set the background-color to black, display to grid, place-items to center, min-height to 100vh (100% of the viewport size), and overflow to hidden.
For our scene div, we just need to set the position to relative.
For our 3 axis divs, we'll give them a height of 1px (1 pixel), a width of 200vh (twice the size of the viewport), a left of -100vh (to center them), and position to absolute. We'll also give each axis their own color, using the x, y, and z classes. We will also rotate y by 90deg around the z-axis, and the z by 90deg around the y axis.
Lastly, we'll give the circle div a position of absolute, border-radius of 50% (which will make it a circle), a width and height of 120px, a left and top of -60px (to center it), and a background of white.
So far, it should look like:

Starting the animations: Rotating the scene

Let's start by animating the entire scene, so it is if we are orbiting the circle. First, we have to set the perspective of the body to 800px, so the "camera" will be 800px from the center. We will also add

  *:not(:empty){
    transform-style: preserve-3d;
  }

to the body which will allow us to see things in 3 dimensions.
Next, let's add an initial rotation around the x-axis by adding transform: rotateX(10deg);.
Since we are rotating the whole scene, we will give the animation to the scene div. Our animation will take 4 inputs: a name, a duration, an iteration number, and a timing function. Let's call this animation "orbit", have it last 20 seconds, repeat infinitely, and with a linear timing function. So far, it should look like animation: orbit 20s infinite linear;
Next, we have to create the keyframes. Each keyframe will specify how far we want the rotation to go. Luckily, we only need two keyframes. The first will be at 0%, where we will give it a transform of rotateY(0). This is the starting position. At 100%, we want it to make a full rotation, so we can set another keyframe at 100%, and give it a transform of rotateY(360deg). We will also have to add the rotateX(10deg) in each transform, in order to keep that constant.
Now, the animation will look like:

Rotating the Circle

Now, let's add an animation to the circle, so it rolls while clicked. First, we have to add the animation to the '.circle' div. Here, we will use animation: roll 5s infinite linear;. We will also use very similar keyframes as the scene, but with rotateX instead of rotateY. If you look at the code now, you'll see that the circle is rotating around the x-axis. However, we only want it to do so while clicked. So in .circle., we will add animation-play-state: paused;. This will set the default state of the circle's animation to not run. Now, we'll add a new section, for .circle:active. This will be applied while the .circle div is being clicked. In here, we will add animation-play-state: running;, making the animation run.
Now, it will be:

Scaling the Circle

Lastly, let's add another animation to the circle.
Firstly, if we just add another animation, it will override the transitions that we already wrote. So let's first wrap our circle div in a new squish div.
Next, we will make it stretch horizontally, then return to normal, then stretch vertically, and then return to normal. Again, we will have to add an animation to .circle. We will add , squish 20s infinite linear to our animation property in .circle. We will also have to add , running to the animation-play-state property, so it will run by default. Now, the keyframes will be more difficult. First, we'll set the keyframes that will be the default values. This will happen at 0%, 50%, and 100%. So we will add a keyframe of 0%, 50%, 100% { transform: scale(1); }. Next, at 25%, we want to have a transform of scaleX(4), which will make it four times as long. Finally, at 75%, we want a transform of scaleX(4) to make it 4 times as tall.
Our final product will be:

Final thoughts

Congrats! This is the first step towards understanding animations in CSS. There are many other operations you can do to make even more complicated animations, and there are some really impressive examples out there. Keep playing around with it make it even cooler!

Irregular Expressions: Matching Strings in Python

Marc Katz — Mon, 03 Jul 2023 17:12:35 +0000

What is Regex?

Regex, short for Regular Expressions, is a way to determine whether a string or a part of a string fits a certain pattern. It is very powerful and short, but the syntax is not very intuitive. In this article, I will give a brief run-down of how to use regex in python, and a basic dictionary for the symbols used.

Startup

In order to use regex in Python, you first need to import the module with import re. Once you've done this, you're good to go!

Search

The first function that we'll go over is search. This function takes in two parameters: the pattern that the function will look for, and the string it'll look for it in. The full syntax is <match> = re.search(<pattern>, <string>). If this function finds a match, it returns a Match object representing the first substring that fits the pattern, otherwise it return None. You can get the matching substring with <match>.group(0), and the start and end positions of the substring as a tuple with <match>.span(). For example, re.search("ra", "abracadabra").span() will return (2,4). If you use groupings in your pattern (see below), you can use <match>.group(<n>), where n is the group number you want to access.

Find All

Findall will give you all the substrings that match the pattern. re.findall(<pattern>, <string>) will return a list of all the non-overlapping substrings in the string that match the pattern. For example, re.findall("a.{1,2}a", "abracadabra") will return ["abra", "ada"]. Note that it doesn't find "aca" nor the second "abra", since they overlap with the substrings already found.

Split

Split is used when you want to break up your original string. re.split(<pattern>, <string>) will return an list of substrings, separated where the pattern matches in the string. For example, re.split("ra", "abracadabra") will return ["ab", "cadab", ""].

Substitute

Sub is used when you want to replace parts of your original string with something else. re.sub(<pattern>, <replacement>, <string>) will replace all the substrings in string that match the pattern with replacement. For example, re.sub("ra", "lo", "abracadabra") will return "ablocadablo".

Special Characters

There are many special characters that can be used in the pattern in order to make your searches more powerful. Here is a list of some of the more widely used ones:

^: Start of the string
$: End of the string
[]: Will match any character inside the square braces. Ranges can be given with -, and ^ will negate it, matching anything except what's inside.

Examples:
- [abc] will match "a", "b", "c"
- [^abc] will match anything except "a", "b", "c"
- [4-7f-e] will match any digit between 4 and 7, and any letter between f and e (inclusive)
.: Wildcard. This will match anything except a newline
\d: Any digit. The same as [0-9]
\D: Any non-digit. The same as [^0-9]
\s: Any whitespace character, such as spaces, tabs, and newlines
\S: Any non whitespace character
\w: Any "word" character: numbers, letters, and _ (underscore)
\W: Any non-"word" character
*: Any number of repetitions of the expression before. For example, a*b will match "b", "ab", "aab", "aaab", etc
+: One or more repetitions of the expression before. For example, a+b will match "ab", "aab", "aaab", etc
?: One or no matches o the expression before. For example, a?b will match "b" and "ab"
{}: Used to match a specific number of repetitions:
- {n}: will match exactly n repetitions:
  - a{3}b will match "aaab"
- {n,}: will match n or more repetitions:
  - a{3,}b will match "aaab", "aaaab", etc
- {n,m}: will match between n and m repetitions, inclusive:
  - a{1,3}b will match "ab", "aab", and "aaab"
</code>: Will escape the next character, allowing you to search for special characters. For example *\? will search for "*?"
|: "Or" function: will match either expression on each side. For example, a|b will match "a" and "b".
(): Will "group" the expression inside the parentheses, either for capturing with the functions above, or to use in relation with the repetition or | symbols. If you don't want to capture, use (?:.

Sources:

https://docs.python.org/3/library/re.html
https://www.w3schools.com/python/python_regex.asp

Useful Links:

https://regex101.com/
https://www.rexegg.com/regex-quickstart.html
https://xkcd.com/1313/
https://alf.nu/RegexGolf

Dark Magic, Evil Bits, and WTF

Marc Katz — Wed, 28 Jun 2023 18:38:59 +0000

How Quake III devs sold their soul to the devil to save a few milliseconds

Content warning: Math

In video games, one of the most computationally intensive function is determining lighting. This is because when calculating light reflections in three dimensions, you need to find 1 divided by the square root of a number, or in other words, 1/sqrt(n), its inverse square root. Since the number will be in floating-point format, division is computationally expensive. With modern computers, this effect is minimal, but back in the 90s, more thought had to be put into each machine cycle to get games to work.

The developers working on Quake III Arena, released in 1999, came up with an ingenious solution to this problem. They found a way to calculate an inverse square root without division. Their code in C, with the original comments, was:

float Q_rsqrt( float number )
{
 long i;
 float x2, y;
 const float threehalfs = 1.5F;

 x2 = number * 0.5F;
 y  = number;
 i  = * ( long * ) &y;                       // evil floating point bit level hacking
 i  = 0x5f3759df - ( i >> 1 );               // what the fuck?
 y  = * ( float * ) &i;
 y  = y * ( threehalfs - ( x2 * y * y ) );   // 1st iteration
// y  = y * ( threehalfs - ( x2 * y * y ) );   // 2nd iteration, this can be removed

 return y;
}

Part 1: Evil Floating point bit level hacking

So, this code is… complicated to say the least, and without knowing what exactly is going on, it is incomprehensible. Even if you know what each line does, why it works is still a long way away.

Let’s start with the line i = * ( long * ) &y;. As commented, this line is evil, in the sense that it breaks the rules. We take y, which is a 32-bit floating-point number, and shove its bits into i, which is a long (a 32-bit integer).

In C, floats are stored in the following fashion: one bit for the sign, eight bits for the exponent (e), and 23 bits for the mantissa (m). If we look at the memory directly, y is 0eeeeeeeemmmmmmmmmmmmmmm (side note: we know that the sign bit will be 0, denoting a positive number, since the square root of a negative number is imaginary). This means that y = (1 + m) * 2 ^ e. Well, kinda. In order to have negative exponents, e is given an offset of 127 (1111111 in binary), so it actually is y = (1 + m) * 2 ^ (e — 127).

So why make it a long? In order to do the following steps, we need to act on y’s bits directly. It is easiest to do this as an integer, since C will now let us add and subtract without worrying about exponents and the like.

Part 2: WTF

Now we get to the next line: i = 0x5f3759df — ( i >> 1 );, and as the developers so succinctly put it in the code’s comments, “what the fuck?”
What are these arrows? What is that number? and how does any of this help us get a square root?

Let’s start with the easiest part first: bit shifting. i >> 1 simply shifts the bits in i once to the right. Since i is a binary number, this is equivalent to dividing by 2. Now, our number looks like 00eeeeeeeemmmmmmmmmmmmmmm. If we view this back as a float, this is approximately equivalent to (m/2) ^ (e/2).

But why do this? Well, we first have to reexamine what we are trying to do. Another way to view 1/sqrt(n) is n^(-1/2). So by bit-shifting right, we’ve (kinda) taken the square root. We negate it by subtracting our new number. But we still have to fix the errors that bit-shifting caused.

The major potential source of error is last bit in the exponent, represented the bold e above, which now has found itself in the area of the mantissa. In order to fix this error, and some other issues (Such as halving the mantissa when we didn’t want to), the developers of Quake III found a Magic Number that helps alleviate these accumulated errors. This magic number is 0x5f3759df, or 1011111001101110101100111011111 in binary, or 1597463007 in decimal. How this number is derived is beyond the scope of this article, but it involves blood sacrifices and dark wizardry (or so I assume). If you want to learn more about this, read Dr. Lomont’s paper, linked below.

And to put this all in a bow, we put our new, probably cursed, number back into a float with the line y = * ( float * ) &i;.

Part 3: Newton’s Method

We’re almost at the finish line, we just need to make our guess (y) more precise. The number we have is good, but it could be better. The way that the code does this is with one iteration of Newton’s method of approximation, x_n+1 = x_n — f(x_n) / f’(x_n), where x_n is the current guess, f(x) is the function we’re trying to approximate, and f’(x) is its derivative. Since Newton’s method finds 0s, we need a function that we want to be 0. In this case, our function will be the error, so our f(y) = 1/y²-x, where x is the number we started with. This means its derivative is f’(y) = -2/y³. Putting this together, we get y — ( (1/y²-x) / (-2/y³) ), which simplifies to y * (1.5 — x * y² / 2). This gives us the final piece of the puzzle, y = y * ( threehalfs — ( x2 * y * y ) );. Now, this method can be done multiple times to further make the it more accurate, but it seems the developers tried a second iteration and found it unnecessary.

And finally, we have our Fast Inverse Square Root. All it took was some hacking and magic. But…

Part 4: Was it worth it?

Yes. This method is way faster than the built-in ways of solving for it, and it is very accurate. The maximum relative error is only 0.175%. But is it worth doing today? Modern computers can handle exponentially more computations than in the 90s, but games and simulations that would requires lots of these calculations have also become more demanding, complex, and exact. Whether or not you want to use a quick approximation like this or just whatever is native to your system will depend on the exact situation you are in and how much computing power you need to squeeze out of your user’s computers.

Sources:
http://www.lomont.org/papers/2003/InvSqrt.pdf
https://betterexplained.com/articles/understanding-quakes-fast-inverse-square-root/