DEV Community: Nicky Marino

Use Pyenv to Streamline your Python Projects

Nicky Marino — Mon, 30 May 2022 00:00:00 +0000

There are two hard problems to solve in a Python project: installing packages, and installing packages in other projects.

This post was originally published on my website. Check it out for more awesome content!

Over the years, the challenges I’ve come across in Python projects has narrowed down to two topics ¹ :

Installing packages for just my current project – not for all projects using that version of Python
Using different Python versions for different projects

Problem (1) can be solved with Python’s built in virtual environments, but they don’t automatically activate when you enter a project folder. As a result, I often forget to run source venv/activate and then accidentally install a bunch of packages to my system Python.

I highly recommend Pyenv to solve both problems. Pyenv is a fantastic tool for managing Python environments simply by changing your current folder. Pyenv also integrates with virtualenv so that you can create virtual environments for any Python version you’d like to use.

Overview

By the end of this post you will know how to:

Install pyenv on your computer
Build a version of Python with pyenv
Change your global and local folder settings to run the new version of Python
Connect pyenv to virtualenv to create a new virtual environment

Install Pyenv

First, install Xcode tools and a few development libraries that the Python build step will need:

# Note: These are macOS specific commands

# Update XCode and Homebrew
xcode-select --install
brew update

# Install libraries you'll need to build Python from source
brew install openssl readline sqlite3 xz zlib

# Install pyenv
brew install pyenv

Note: The installation instructions for pyenv change frequently. These are the steps for pyenv==2.3.1 on macOS. If you are using a different version or OS, see the pyenv installation guide.

If you’re using a zsh shell, run these commands to add pyenv to your dot files. If you’re using another shell, follow the installation steps for your shell under the Shell environment section of the pyenv README.

echo 'export PYENV_ROOT="$HOME/.pyenv"' >> ~/.zshrc
echo 'command -v pyenv >/dev/null || export PATH="$PYENV_ROOT/bin:$PATH"' >> ~/.zshrc
echo 'eval "$(pyenv init -)"' >> ~/.zshrc

Restart your terminal for the changes to take effect, and verify your installation by entering the command which python3 and confirming that pyenv/.shims is somewhere in the output:

$ which python3

/Users/nicky/.pyenv/shims/python3

Install Python 3.8.2

Next let’s install Python 3.8.2. I chose this particular version of Python because I use it in a few of my side projects, so you can use any other version of Python if you’d like.

Recently, I came across a bug in macOS Big Sur that prevents libraries like numpy and pandas to use the lzma package. Credit to the thread for pyenv Issue #1737 for the below instructions to correct the 3.8.2 build.

To install Python 3.8.2, first reinstall the zlib and bzip2 libraries via Homebrew:

brew reinstall zlib bzip2

In your ~/.zshrc (or ~/.bashrc if you’re using bash), add the below flags for the Python compiler to read:

export LDFLAGS="-L/usr/local/opt/zlib/lib -L/usr/local/opt/bzip2/lib"
export CPPFLAGS="-I/usr/local/opt/zlib/include -I/usr/local/opt/bzip2/include"

Then install Python 3.8.2 using the below command. This may take some time to compile.

CFLAGS="-I$(brew --prefix openssl)/include -I$(brew --prefix bzip2)/include -I$(brew --prefix readline)/include -I$(xcrun --show-sdk-path)/usr/include" \
LDFLAGS="-L$(brew --prefix openssl)/lib -L$(brew --prefix readline)/lib -L$(brew --prefix zlib)/lib -L$(brew --prefix bzip2)/lib" \
pyenv install --patch 3.8.2 < <(curl -sSL https://github.com/python/cpython/commit/8ea6353.patch\?full_index\=1)

Verify that you built and installed Python 3.8.2 by checking what versions of Python pyenv has:

$ pyenv versions
  system
  3.8.2

Manage Pyenv Versions

Pyenv has two types of versions:

local – the Python version for your current directory
global – the default Python version to use if no local version is set.

When you switch directories in your terminal, Pyenv checks for a .python-version file in the root of that directory. If there’s no .python-version, Pyenv will search each parent folder until one is found.²

You can set what version of Python pyenv will use in your current folder with pyenv local:

$ pyenv local 3.8.2

You can check .python-version to see what version is used:

$ cat .python-version
3.8.2

And pyenv version will also tell you what version of Python you’re currently using as well as what config file set it:

$ pyenv version
3.8.2 (set by /Users/nicky/Developer/my-project/.python-version)

If you move to a different folder without a .python-version, Pyenv will use your global setting, which is system by default:

$ cd /path/to/other/folder

$ pyenv version
system (set by /Users/nicky/.python-version)

How to Use Pyenv Virtual Environments

Give pyenv superpowers by installing pyenv-virtualenv. With this plugin, pyenv can manage virtual environments like venv and Conda environments.

Install the plugin with Homebrew:

brew install pyenv-virtualenv

And add the following to your shell’s .rc file (such as .zshrc or .bashrc):

eval "$(pyenv virtualenv-init -)"

To create a virtual environment for the Python version you’re using with pyenv, run pyenv virtualenv [version] [new environment name]. For example, to create a new venv for a sample project using 3.8.2:

pyenv virtualenv 3.8.2 my-project-3.8.2

While not required, I recommend adding the Python version to your environment names to manage them easier.

You can also create a new virtual environment using the current pyenv Python version:

pyenv virtualenv my-default-venv

Just like pyenv, your virtual environment will automatically activate whenever you move into that folder!

Helpful Commands

Below is a list of commands for pyenv that I use often.

To show all of your Python versions and all virtual environments:

pyenv versions

To set a different version for the python and python3 commands for your global settings:

# (This works the same with `local` as well)
pyenv global [python version] [python3 version]

To create a new virtual environment and use it locally:

pyenv virtualenv [python version] [new environment name]
pyenv local [new environment name]

The rest are python vs python3 compatibility problems. ↩
The .python-version at the root of your home folder sets your global Python version. ↩

Add Your Vimrc to Obsidian

Nicky Marino — Tue, 22 Mar 2022 02:03:32 +0000

I’ve migrated my note taking system from Craft to Obsidian over the past few weeks, and it’s been great so far. One major advantage of Obsidian is its amazing developer community and wide, wide universe of community plugins. I use quite a few community plugins:

One of my favorite features of Obsidian is its support for vim bindings. I use vim and VS Code’s vim bindings all the time, and I’m able to context switch much faster between coding and note writing when all of my editors use vim keys.¹

You can turn on vim bindings in your own Obsidian vault by going to “Settings” > “Editor” and toggling “Vim key bindings”:

The only downside to Obsidian’s default vim bindings is the lack of .vimrc support out of the box. My personal .vimrc has a few shortcuts that I rely on:

I map ; to : in normal mode so that I don’t rely on the shift key
I map j to gj (and k to gk) to jump by visual lines instead of logical lines by default

The ; mapping isn’t a deal breaker in Obsidian; I’m not using vim commands that often. However, the j and k remaps are critical! Obsidian wraps text much more than a terminal thanks to its generous padding. Even on this “simple” post with just a lot of text, one line turns into four in Obsidian:

Luckily, there’s plugin for Obsidian .vimrc files: Obsidian Vimrc Support Plugin. To add this to your Obsidian vault, go to “Settings” > “Community plugins” > “Browse” and search for “vimrc”:

Then click “Vimrc Support”, “Install” and “Enable”:

At the root of your Obsidian vault (the root folder for all of your .md files), create a new file named .obsidian.vimrc. You can paste any of your ~/.vimrc into your .obsidian.vimrc.

Here’s my .obsidian.vimrc that remaps ;, j, and k in normal mode:

" .obsidian.vimrc
"
" A small .vimrc for Obsidian vim bindings
"
" To enable this file, you must install the Vimrc Support plugin for Obsidian:
" https://github.com/esm7/obsidian-vimrc-support
" _________________________________________________________________________

" ; (semicolon) - same as : (colon)
nmap ; :

" (space) - same as : (colon)
nmap <SPACE> :

" j and k navigate visual lines rather than logical ones
nmap j gj
nmap k gk

Once you write your .obsidian.vimrc, reload Obsidian and your new vim bindings will load!

The only downside is that I start typing gibberish whenever I open Google Docs or Word, but that’s a sacrifice I’m willing to make. ↩

How to Install Ruby 2.7.3 on M1 Mac

Nicky Marino — Fri, 17 Dec 2021 00:00:00 +0000

This post was originally published on my website. Check it out for more awesome content!

Installing Ruby or Python on M1 Macs is a nightmare. I’ve lost so many hours fighting compilers and Rosetta to these issues, so I’m documenting my installation steps to spare you¹ a lot of headache.

Why 2.7.3?

First, why Ruby 2.7.3 specifically? Ruby 3.0.0+ works great on M1 using my RVM install instructions post:

rvm install 3.0.0

Unfortunately, my website’s host GitHub Pages uses Ruby 2.7.3 according to their dependency documentation. And a bare-bones install of 2.7.3 blows up on my M1 machine:

rvm install 2.7.3

So I needed to find a way to install 2.7.3 on my Mac to build and run my website locally to preview new posts.

The Solution

I’ll put the winning command here at the top to keep things simple. Many thanks to @d-lebed for documenting their solution on this GitHub issue. I lightly modified their code to use $(brew --prefix) instead of hardcoding where homebrew downloads openssl@1.1:

# Winning script!
brew install openssl@1.1

export PATH="$(brew --prefix)/opt/openssl@1.1/bin:$PATH"
export LDFLAGS="-L$(brew --prefix)/opt/openssl@1.1/lib"
export CPPFLAGS="-I$(brew --prefix)/opt/openssl@1.1/include"
export PKG_CONFIG_PATH="$(brew --prefix)/opt/openssl@1.1/lib/pkgconfig"

rvm autolibs disable

export RUBY_CFLAGS=-DUSE_FFI_CLOSURE_ALLOC
export optflags="-Wno-error=implicit-function-declaration"

rvm install 2.7.3 --with-openssl-dir=$(brew --prefix)/opt/openssl@1.1

This successfully installed Ruby 2.7.3 for me, and I was then able to run bundle install at the root of my website’s repo!

The Issue

I originally documented all of my problem solving steps and install attempts, but it grew too long and rambly for my liking. Instead, I’ll give a brief summary of a few errors I came across.

Flags

At one point, rvm was complaining about my LDFLAGS:

checking whether LDFLAGS is valid... no
configure: error: something wrong with LDFLAGS="-L/usr/local/opt/zlib/lib -L/usr/local/opt/bzip2/lib"

And setting LDFLAGS="" in front of rvm install 2.7.3 only resulted in errors with the compiler missing openssl libraries. So I needed LDFLAGS set somehow, but not the way that works for Python and pyenv.

Rosetta

In a few GitHub issues, people recommended opening the Terminal app via Rosetta and running commands that way such as in this issue comment. However, I saw no difference in error outputs between Rosetta Terminal and iTerm. In the end, I used iTerm without Rosetta to successfully install Ruby 2.7.3.

Usename Macro Error

I got close to a correct install when I added the openssl library to LDFLAGS, CPPFLAGS, and PKG_CONFIG_PATH:

PATH="/usr/local/opt/openssl@1.1/bin:$PATH" \
LDFLAGS="-L$(brew --prefix)/opt/openssl@1.1/lib" \
CPPFLAGS="-I$(brew --prefix)/opt/openssl@1.1/include" \
PKG_CONFIG_PATH="$(brew --prefix)/opt/openssl@1.1/lib/pkgconfig" \
arch -x86_64 rvm install 2.7.3 -j 1

But I then started seeing errors about some missing rl_username_completion_function macro:

214 warnings generated.
linking shared-object date_core.bundle
installing default date_core libraries
compiling readline.c
readline.c:1904:37: error: use of undeclared identifier 'username_completion_function'; did you mean 'rl_username_completion_function'?
                                    rl_username_completion_function);
                                    ^ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
                                    rl_username_completion_function
readline.c:79:42: note: expanded from macro 'rl_username_completion_function'
# define rl_username_completion_function username_completion_function
                                         ^
/opt/homebrew/opt/readline/include/readline/readline.h:485:14: note: 'rl_username_completion_function' declared here
extern char *rl_username_completion_function PARAMS((const char *, int));
             ^
1 error generated.
make[2]: *** [readline.o] Error 1
make[1]: *** [ext/readline/all] Error 2
make: *** [build-ext] Error 2
+__rvm_make:0> return 2

I then found the winning command (above) by googling for rvm install 2.7.3 error extern char *rl_username_completion_function PARAMS((const char *, int)); and trying a few commands recommended on some GitHub issues.

Next Steps

I recently figured out how to install Python 3.7, 3.8, and 3.9 on M1 Macs via pyenv, so keep an eye out for my post on how to install those.

And future me ↩

A Beginner's Guide to Flask and Replit

Nicky Marino — Tue, 13 Apr 2021 00:00:00 +0000

This post was originally published on my website. Check it out for more awesome content!

I originally wrote this tutorial for Code Your Dreams, an incubator of youth voice, tech skills, and social justice. Their project-based and student-centered programs enable youth to be the change makers we know they are through code. To find out how to get involved, visit their website: https://www.codeyourdreams.org

Replit is a free, collaborative, in-browser IDE for creating new projects without setting up any environments on your computer. With Replit, you don’t need to “deploy” your projects to any service; they’ll be instantly available to you as soon as you start typing. In this post, we’ll review how to create a Flask app, set up folders for HTML and CSS templates, and learn how to navigate your Flask app.

Before following these steps, you must first create an account on replit.

Creating a Flask Project

First, let’s create a blank Python project. On your replit homepage, create new project by clicking “Python” under the “Create” heading:

For the project name, type my-first-flask-site and click “Create repl”:

Your new project will automatically create a file named main.py and open a Python IDLE for you, but we need to install Flask before we can start writing our app. On the left sidebar, click the “Packages” icon, which looks like a hexagonal box:

From here, we can install any Python packages that you want to import in your app. Install the Flask package by typing “flask” and selecting the first item from the list named “Flask”:

Then, click the “Files” icon on the left sidebar to go back to the files list. You should see main.py, which was already created for you.

Hello, World!

Our first Flask app will have one page—the index page—that says Hello World! when we go to the home page. Copy the below code into the file named main.py:

from flask import Flask, render_template

# Create a flask app
app = Flask(
  __name__ ,
  template_folder='templates',
  static_folder='static'
)

# Index page
@app.route('/')
def hello():
  return "Hello World!"

if __name__ == ' __main__':
  # Run the Flask app
  app.run(
    host='0.0.0.0',
    debug=True,
    port=8080
  )

In this code, we have one page that’s controlled by the hello() function. It’s route is ’/‘, which means that it is at the home page of our app.

For flask projects, Replit looks for a web server at the local URL http://0.0.0.0:8080, so we need set the host to ’0.0.0.0' and the port to 8080 in app.run(…). We also set debug=True so that any changes you make to files will be automatically updated when you refresh a page.

We’ll use render_template, template_folder, and static_folder later in this tutorial, so don’t worry about those just yet.

Now, click the green “Run” button at the top of the page. Replit should install Flask, then open a browser with your first Flask app!

The bottom right window is the Python console, and will show any error messages or logs that are printed.

At any time, you can click the “Stop” button at the top and click “Run” again to restart your Flask app.

Sometimes, your app might have multiple pages. To go to a different URL (or @app.route) in your app, click the icon “Open in a new tab” on the browser window. It will be to the right of the address bar:

You can type in a new path in the address bar of the new tab, such as https://my-first-flask-site.YOUR_USERNAME.repl.co/my-second-page. Right now, your server will return a 404 for that page because it doesn’t exist.

HTML Assets and CSS Styles

Next, let’s add an HTML file and a CSS file to our Flask app. HTML files are commonly put in a templates folder in a Flask project, because they are usually templates to show information. Our Flask code will supply values to HTML templates via variables, so that our app can change via Python code.

Adding an Index Page

In main.py, we already set up our Flask app to look in the templates folder for HTML files:

# Create a flask app
app = Flask(
  # ...
  template_folder='templates',
  # ...
)

Now let’s create the templates folder and create an index.html file. Next to the “Files” header on the top left, click the “Add folder” button and name the new folder templates. Then, click the three dot icon on the templates folder and click “Add file”. Name the new file index.html. To have both index.html and main.py open at the same time in your editor, right click main.py on the files list and click “Open tab”.

Your editor should now look like this (note that there’s two tabs in the editor now: one for templates/index.html and one for main.py):

Copy the below code into the templates/index.html file:

<!doctype html>
<head>
    <title>My First Flask Website</title>
    <link href="/static/style.css" rel="stylesheet" type="text/css">
</head>
<body>
    <h1>Hello, World!</h1>
    <p>
        Welcome to your first Flask website
    </p>
</body>

And replace your contents in main.py with the below code. This new version updates the hello() function to index(), and it returns the contents of index.html to the user:

from flask import Flask, render_template

# Create a flask app
app = Flask(
  __name__ ,
  template_folder='templates',
  static_folder='static'
)

# Index page (now using the index.html file)
@app.route('/')
def index():
  return render_template('index.html')

if __name__ == ' __main__':
  # Run the Flask app
  app.run(
    host='0.0.0.0',
    debug=True,
    port=8080
  )

Click the Refresh button in the browser window of the project ( not the refresh button in Chrome or Firefox, but the refresh button for the smaller window in your project), and you should see the contents of your index page with a large header:

Adding CSS Styling

Now let’s add a CSS file to change the color of the text of our app. The Flask app is set up to look inside a static folder for CSS and JS assets:

# Create a flask app
app = Flask(
  # ...
  static_folder='static'
)

And the index page is set up to look for a file named style.css inside the static folder:

<!doctype html>
<head>
    <!-- ... -->
    <link href="/static/style.css" rel="stylesheet" type="text/css">
    <!-- ... -->

Click main.py on the Files list, then click the “Add a folder” icon to the left of the “Files” header. Name the new folder static. Next, click the three dot icon next to your new static folder and click “Add file”. Name the file style.css, and open it by right clicking the file and selecting “Open in tab”.

Your project should now look like this:

Let’s write some CSS to change the color of the “Welcome” message to red. Add the following code to your static/style.css file:

p {
  color: red;
}

Click the Refresh button in the project browser window ( not the refresh button in Chrome or Firefox, but the refresh button for the smaller window in your project), and the “Welcome” screen will turn red:

Congratulations, you’ve written your very first Flask app!

How to Draft Blog Posts in Notion

Nicky Marino — Fri, 20 Nov 2020 00:00:00 +0000

This post was originally published on my website. Check it out for more awesome content!

Maintaining a streamlined process to draft, edit, and publish technical articles can be tricky. For example, I write across multiple topics such as Python tutorials, bash commands, and Ruby optimizations. I also publish to three different locations—my website, dev.to, and Hashnode, so I write my posts in Markdown-friendly tools for cross compatibility.

Notion is a great tool for drafting blog posts, technical or otherwise. Notion has an entire suite of tools to make drafting easy: Markdown exports, page properties and filters, a beautiful interface, and many more. With Notion, I quickly draft articles, publish each one across multiple sites, and monitor each article’s progress at a glance. Here’s what my setup looks like:

In this article, we will walk through creating an Articles page in Notion. We’ll create:

A page for hold articles
An articles database
A new draft template, and
A Kanban board to display each draft’s status

Markdown, Jekyll, and Drafts! Oh My!

For a while, I kept my drafts in a _drafts folder in my website’s repo. Jekyll, my static site generator of choice, ignores any Markdown files in the _drafts folder. However, I’ve been hesitant to heavily draft and edit posts in a publicly available location, so I haven’t been drafting many posts lately. I love that my website is open source for others to use as an example for static site layouts, but I would like to keep the messiness of article writing to myself.

Then, I tried drafting my articles in the app Drafts. Using Drafts, I could write posts in Markdown and preview them privately, but the only way to mark the status of a post (draft, edit, published, abandoned, etc.) was via tags. I used tags like blog and draft for quick filtering:

Unfortunately, tags didn’t scale for me. The only way to view each draft is from the small sidebar on the left, and I had to remember what combination of tags I used to mark blog drafts. Often, I’d forget to tag posts, or I’d forget to update the tags after I had drafted it. I also wanted to have a space devoted only to my articles so that I could get a bird’s eye view of what I had in the pipeline. That’s where Notion databases come to the rescue.

The Notion Database

Databases in Notion let me organize and structure related pages. Now that my articles are in a database, I can label, search, and create views of articles. For example, I can look at all of my articles as a list:

I can also view article drafts as a gallery, with a preview of each article:

I can even view my articles as a Kanban board based on their status. I use this birds eye view to see what articles are currently being drafted, which are in review, and which have been posted or abandoned:

Creating the Table

Let’s start by creating a page and a database to hold the articles. First, click “+ Add a page” on the left sidebar. Give the page any title, icon, and header that you want. I titled mine “My Articles”:

Next, we need to create an article database. Click the text that says “Type ‘/’ for commands”, then type /list inline. Choose the option labelled “List - Inline” under “Database”:

Notion will create a new list for you and populate a few example pages:

Add a title to your table by clicking on “Untitled” and adding a title, such as “Articles”:

Table Properties

Let’s add some article properties for our pages. In this database, each article will be a page in Notion. Any properties we add to the Articles database will be shown in each page. Each article needs a title, creation date, last edited time, and status. For articles, the title property will be the the title of the Notion page, but we need to add the rest of the properties to the table.

To add properties, hover over the Articles table, click on the three dots on the right, and select “Properties”. Add a status property by clicking “+ Add a property”. Then, for “Property Type”, pick “Select”. To show the property in the table view, enable the toggle next to Select. Similarly, add a “Created” property by selecting the type “Created time”, and add an “Edited” property by selecting the type “Last edited time”. Delete the “Tags” property by selecting the dots on the left, and then select “Delete”. The Articles table properties now look like this:

Now that our table is set up for articles, we can create a “New Draft” page template!

The Page is as Mighty as the Table

Most likely, you’ll add new pages to your Articles table when you’re creating a new draft. With a “New Draft” page template, it’s easy to start drafting and not need to worry about setting any page properties. To create a new draft, click the dropdown arrow next to “New” on the Articles table, and then click “+ New template”.

Name the template “New Draft”. We’ll want any new draft to have a Draft status, so we need to create that status. Click the text “Empty” next to the Status property, type in “Draft”. This will create the Draft status for any Article page, and assign the Draft status to any page that is created with the New Draft template. I also added a ✏️ icon to the New Draft template and the Draft that they’re easy to find in lists:

Hello World

Let’s write our first article draft! Click the “Back” button to go back to the Articles table, then select “New Draft” to begin writing. Add a title and some example text for your first draft:

Click “Back” again to go to the Articles table, and delete the old example pages. Practice using Notion page templates by adding another draft:

You are Your Own Publisher

One way to publish your articles is to make the page publicly accessible in Notion. You can also use Notion’s Markdown exporter to convert a page into Markdown and then import the content into sites such as dev.to or Hashnode. To export a page to Markdown, click the three dots in the top right of a page, select “Export”, then select “Markdown & CSV”. The page contents will be converted into Markdown for cross compatibility:

Great, Now There’s Two of Them

You’ve finished editing and publishing your “Hello, World!” article. Now it’s time to change the status of that article to Posted, but we don’t have an easy way to do that yet. Let’s create a new Kanban board view of our Articles table. With a Kanban board, you can move articles as cards from one status to the next.

To create a new table view, click “+ Add a view” at the top of the Articles table. Select the “Board” option and then click “Create”. Notion will automatically set up the Board view to sort articles by the Status property:

To add the Posted status, click “+ Add a group” and enter “Posted”. No pages have the Posted status yet, so the column will be empty:

To mark the “Hello, World!” article as posted, simply drag that article’s card from Draft to Posted, and it’s done!

With Notion’s databases, page properties, and custom views, you can create your own system to keep track of your blog posts and articles. I’m looking forward to using Notion more to structure my data in different aspects of my life.

What are other ways you've used Notion to keep track of your life? Leave a comment and let me know!

How to Write List Comprehensions with Python

Nicky Marino — Tue, 29 Sep 2020 00:00:00 +0000

This post was originally published on my website. Check it out for more awesome content!

One of the most common blocks you will write in Python scripts is a for loop. With for loops, you can repeat the same set of instructions in a block over and over. Python’s for loops are really foreach loops, where you repeat the instructions for every item in a collection. These collections are called iterators, which is something that a Python loop is able to iterate over, and the most common iterator is list.

For Loops

Let’s look at an example of a for loop. Write a function that prints the square of each number from one to n:

def write_squares(n):
    for number in range(n):
        square = number ** 2
        print(square)

write_squares(6)

This is the output you will get from the above example:

In write_squares, we calculate the variable square and print it for each value in range(n), which is all of the numbers from 0 to n inclusive.

Small for loops like this are very common in Python scripts. For example, we can read in lines from a file and strip any spaces from each line:

def lines_from(filename):
    with open(filename, "r") as file_obj:
        original_lines = file_obj.readlines()

    stripped_lines = []
    for line in original_lines:
        new_line = line.strip()
        stripped_lines.append(new_line)

Or, we might have a list of IDs and want to grab a piece of data for each ID:

def email_addresses(ids):
    addresses = []
    for identifier in ids:
        data = api_request(identifier)
        email_address = data["email"]
        addresses.append(email_addresses)

With each of these for loops, we’re running the same piece of code on each item in the list. While the repeated piece of code isn’t too complicated, you still need to write (or read!) multiple lines to understand what the code block is doing. List comprehensions are an elegant way to create lists from existing lists.

List Comprehension

First, let’s look at a quick example:

chars = [letter for letter in "Hello, world!"]
print(chars)

When you run this Python code, the output will be:

['H', 'e', 'l', 'l', 'o', ',', ' ', 'w', 'o', 'r', 'l', 'd', '!']

In this example, a new list is created, named chars, that contains each of the items in the string "Hello, world!". Because items in a string are characters, chars is a list of every character in the string.

List Comprehension Syntax

List comprehensions are written as:

new_list = [expression for item in old_list]

This is what a list comprehension “unwrapped” into a traditional list would look like:

new_list = []
for item in old_list:
    new_item = expression
    new_list.append(new_item)

Used correctly, list comprehensions can reduce for loops into a more readable, one line expression.

Examples

Let’s re-write our earlier examples using list comprehensions. Instead of writing an whole new function, our write_squares example can be reduced to one line:

squares = [number ** 2 for number in range(6)]
print(squares)

We get the same results when we run this code:

Now let’s look at the line stripping function. We need to keep the first few lines to read the file contents, but the for loop that appended each stripped line to a new variable has been updated to use a list comprehension instead:

def lines_from(filename):
    with open(filename, "r") as file_obj:
        original_lines = file_obj.readlines()

    return [line.strip() for line in original_lines]

The email address fetcher can be reduced to one line, and it can be directly placed in another block of code:

def email_addresses(ids):
    return [api_request(id)["email"] for id in ids]

Conditionals

List comprehensions can also use conditional statements to filter or modify the values that are added to a new list. For example, if we wanted a list of only even integers from 1 to 20, we could add a conditional to the end of a list comprehension:

>>> evens = [num for num in range(0, 20) if num % 2 == 0]
>>> print(evens)

[0, 2, 4, 6, 8, 10, 12, 14, 16, 18]

List comprehensions can use any number of and and or operators in conditionals. For example, we can use the conditional (num % 2 == 0) and (num % 3 == 0) to keep only numbers that are divisible by both 2 and 3:

>>> my_nums = [num for num in range(0, 20) if (num % 2 == 0) and (num % 3 == 0)]
>>> print(my_nums)

[0, 6, 12, 18]

Key Points

List comprehension is an elegant way to create new lists from existing lists. List comprehensions can reduce multiple-line code blocks to just one line! However, avoid writing large list comprehensions, as that may reduce legibility for your code readers.

Once you’re comfortable with list comprehensions, I recommend learning dictionary comprehensions, which are very similar to list comprehensions except that they operate on dictionaries.

Python's Walrus Operator

Nicky Marino — Wed, 23 Sep 2020 00:00:00 +0000

This post was originally published on my website. Check it out for more awesome content!

Beautiful is better than ugly.

— The Zen of Python

Python introduced a brand new way to assign values to variables in version 3.8.0. The new syntax is :=, and it’s called a “walrus operator” because it looks like a pair of eyes and a set of tusks. The walrus operator assigns values as part of a larger expression, and it can significantly increase legibility in many areas.

Named Expressions

You can create named expressions with the walrus operator. Named expressions have the format NAME := expression, such as x := 34 or numbers := list(range(10)). Python code can use the expression to evaluate a larger expression (such as an if statement), and the variable NAME is assigned the value of the expression.

If you’ve written Swift code before, Python’s walrus operator is similar to Swift’s Optional Chaining. With optional chaining, you assign a value to a variable inside the conditional of an if statement. If the new variable’s value is not nil (like Python’s None), the if block is executed. If the variable’s value is nil, then the block is ignored:

let responseMessages = [
    200: "OK",
    403: "Access forbidden",
    404: "File not found",
    500: "Internal server error"
]

let response = 444
if let message = responseMessages[response] {
    // This statement won't be run because message is nil
    print("Message: " + message)
}

Benefits

There are a lot of benefits to using the walrus operator in your code. But don’t take my word for it! Here’s what the authors of the idea said in their proposal:

Naming the result of an expression is an important part of programming, allowing a descriptive name to be used in place of a longer expression, and permitting reuse.

— PEP 572 – Assignment Expressions

Let’s take a look at some examples.

Don’t Repeat Yourself

With the walrus operator, you can more easily stick to the DRY principle and reduce how often you repeat yourself in code. For example, if you want to print an error message if a list is too long, you might accidentally get the length of the list twice:

my_long_list = list(range(1000))

# You get the length twice!
if len(my_long_list) > 10:
    print(f"List is too long to consume (length={len(my_long_list)}, max=10)")

Let’s use the walrus operator to only find the length of the list once and keep that length inside the scope of the if statement:

my_long_list = list(range(1000))

# Much better :)
if (count := len(my_long_list)) > 10:
    print(f"List is too long to consume (length={count}, max=10)")

In the code block above, count := len(my_long_list) assigns the value 1000 to count. Then, the if statement is evaluated as if len(my_long_list) > 10. The walrus operator has two benefits here:

We don’t calculate the length of a (possibly large) list more than once
We clearly show a reader of our program that we’re going to use the count variable inside the scope of the if statement.

Reuse Variables

Another common example is using Python’s regular expression library, re. We want to look at a list of phone numbers and print their area codes if they have one. With a walrus operator, we can check whether the area code exists and assign it to a variable with one line:

import re

phone_numbers = [
    "(317) 555-5555",
    "431-2973",
    "(111) 222-3344",
    "(710) 982-3811",
    "290-2918",
    "711-7712",
]

for number in phone_numbers:
    # The regular expression "\(([0-9]{3})\)" checks for a substring
    # with the pattern "(###)", where # is a 0-9 digit
    if match := re.match("\(([0-9]{3})\)", number):
        print(f"Area code: {match.group(1)}")
    else:
        print("No area code")

Legible Code Blocks

A common programming pattern is performing an action, assigning the result to a variable, and then checking the result:

result = parse_field_from(my_data)
if result:
    print("Success")

In many cases, these types of blocks can be cleaned up with a walrus operator to become one indented code block:

if result := parse_field_from(my_data):
    print("Success")

These blocks can be chained together to convert a nested check statements into one line of if/elif/else statements. For example, let’s look at some students in a dictionary. We need to print each student’s graduation date if it exists, or their student id if available:

sample_data = [
    {"student_id": 200, "name": "Sally West", "graduation_date": "2019-05-01"},
    {"student_id": 404, "name": "Zahara Durham", "graduation_date": None},
    {"student_id": 555, "name": "Connie Coles", "graduation_date": "2020-01-15"},
    {"student_id": None, "name": "Jared Hampton", "graduation_date": None},
]

for student in sample_data:
    graduation_date = student["graduation_date"]
    if graduation_date:
        print(f'{student["name"]} graduated on {graduation_date}')
    else:
        # This nesting can be confusing!
        student_id = student["student_id"]
        if student_id:
            print(f'{student["name"]} is currently enrolled with ID {student_id}')
        else:
            print(f'{student["name"]} has no data")

With walrus operators, we can put the graduation date and student id checks next to each other, and better show that we’re checking for one or the other for each student:

sample_data = [
    {"student_id": 200, "name": "Sally West", "graduation_date": "2019-05-01"},
    {"student_id": 404, "name": "Zahara Durham", "graduation_date": None},
    {"student_id": 555, "name": "Connie Coles", "graduation_date": "2020-01-15"},
    {"student_id": None, "name": "Jared Hampton", "graduation_date": None},
]

for student in sample_data:
    # Much cleaner
    if graduation_date := student["graduation_date"]:
        print(f'{student["name"]} graduated on {graduation_date}')
    elif student_id := student["student_id"]:
        print(f'{student["name"]} is currently enrolled with ID {student_id}')
    else:
        print(f'{student["name"]} has no data")

Wrap Up

With walrus operators and named expressions, we can dramatically increase the legibility of our code by simplifying statements, reusing variables, and reducing indentation. For more great examples, check out the original proposal and the Python 3.8 release notes.

Multiple Named Profiles with AWS CLI

Nicky Marino — Sun, 06 Sep 2020 22:30:00 +0000

This post was originally published on my website. Check it out for more awesome content!

The AWS CLI supports named profiles so that you can quickly switch between different AWS instances, accounts, and credential sets. Let's assume you have two AWS accounts, each with an access key id and a secret access key. The first account is your default profile, and the second account is used less often.

Adding a Named Profile

First, open ~/.aws/credentials (on Linux & Mac) or %USERPROFILE%\.aws\credentials (on Windows) and add your credentials:

[default]
aws_access_key_id=AKIAIOSFODNN7EXAMPLE1
aws_secret_access_key=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY1

[user2]
aws_access_key_id=AKIAI44QH8DHBEXAMPLE2
aws_secret_access_key=je7MtGbClwBF/2Zp9Utk/h3yCo8nvbEXAMPLEKEY2

If your two profiles use different regions, or output formats, you can specify them in ~/.aws/config (on Linux & Mac) or %USERPROFILE%\.aws\config (on Windows):

[default]
region=us-west-2
output=json

[profile user2]
region=us-east-1
output=text

Note: do not add profile in front of the profile names in the credentials file, like we do above in the config file.

Most AWS CLI commands support the named profile option --profile. For example, verify that both of your accounts are set up properly with sts get-caller-identify:

# Verify your default identity
$ aws sts get-caller-identity

# Verify your second identity
$ aws sts get-caller-identity --profile user2

EKS and EC2 commands also support the --profile option. For example, let's list our EC2 instances for the user2 account:

$ aws ec2 describe-instances --profile user2

Setting a Profile for Kubeconfig

The AWS CLI --profile option can be used to add new clusters to your ~/.kubeconfig. By adding named profiles, you can switch between Kubernetes contexts without needing to export new AWS environment variables.

If your EKS instance is authenticated with only your AWS access key id and access key secret, add your cluster with eks update-kubeconfig:

$ aws eks update-kubeconfig --name EKS_CLUSTER_NAME --profile PROFILE

If your EKS instance uses an IAM Role ARN for authentication, first copy the role ARN from the AWS Console: Go to the EKS service page, then Clusters, then select your cluster name, and find the IAM Role ARN at the bottom of the page. The format of the role ARN is typically arn:aws:iam::XXXXXXXXXXXX:role/role_name. Then, use eks update-kubeconfig:

aws eks update-kubeconfig --name EKS_CLUSTER_NAME --role-arn ROLE_ARN --profile PROFILE

To verify that your kubeconfig is set properly, use kubectx to switch to one of your new clusters and try to list out its services:

$ kubectx EKS_CLUSTER_NAME
Switched to context "EKS_CLUSTER_NAME".

$ kubectl get services
...

How to Use Jekyll on macOS Catalina with RVM

Nicky Marino — Sat, 13 Jun 2020 00:00:00 +0000

This post was originally published on my website. Check it out for more awesome content!

Apple bundles a system version of the Ruby programming language on macOS. Because system Ruby is used by the inner workings of the operating system, this version is not meant to be upgraded or modified by a user. With the Ruby Version Manager RVM, you can install an additional Ruby version for personal use.

Similar to pyenv, you can install multiple versions of Ruby with RVM and change the version you’re using on the fly. You can also install gems without sudo.

Installing RVM and Ruby

Before downloading RVM, first install gpg and the mpapis public key:

$ brew install gnupg
$ gpg --keyserver hkp://ipv4.pool.sks-keyservers.net --recv-keys xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

The keys (xxxx...) change often, so you will need to copy the most recent ones from the RVM install page.

Next, download the most recent stable version of RVM:

$ \curl -sSL https://get.rvm.io | bash -s stable --ruby

After installation, RVM will tell you to either open a new terminal or source rvm, so run the command it prints:

$ source ~/.rvm/scripts/rvm

You will also want to add rvm to your ~/.zshrc or ~/.bashrc to load when you open a terminal:

# Add this to your ~/.zshrc or ~/.bashrc
[[-s "$HOME/.rvm/scripts/rvm"]] && . "$HOME/.rvm/scripts/rvm"

Use rvm list to find a Ruby version you want to install, then tell RVM which version to use:

$ rvm list
$ rvm use 2.7.0

You can then verify that you’re using an RVM-managed version of Ruby:

$ which ruby
~/.rvm/rubies/ruby-2.7.0/bin/ruby

Installing Jekyll

First, verify you’re using a Ruby version managed by RVM in the above step. Then, install the Jekyll gem:

$ gem install jekyll bundler

If you’re already in a Jekyll website repo (or any folder with a Rakefile), you can use bundle to install your remaining requirements:

$ bundle install

You may then need to update Jekyll for your Rakefile requirements:

$ bundle update jekyll

Now you can up the Jekyll server:

$ bundle exec jekyll serve

Now check out your site at http://localhost:4000! See the Jekyll Quickstart for more details on starting a Jekyll blog.

Optimizing a Ruby Script with NArray

Nicky Marino — Tue, 21 Jan 2020 00:00:00 +0000

This post was originally published on my website. Check it out for more awesome content!

Earlier this week, I had released Virgo, a Ruby CLI to generate wallpapers (including the one above). My goal was to be able to create beautiful OLED wallpapers for my phone, but unfortunately, the first version of Virgo would take about 15 seconds to generate a wallpaper the size of an iPhone 11. The first version of Virgo used ChunkyPNG::Image to place pixels on the background, and the author of ChunkyPNG alludes to this possible problem in his README:

Also, have a look at OilyPNG which is a mixin module that implements some of the ChunkyPNG algorithms in C, which provides a massive speed boost to encoding and decoding.

Improvement Goal

My goal was to reduce the time to generate an iPhone-sized wallpaper (about 1200 by 2200 pixels) in under one second. I chose this constraint so that I can eventually write a web frontend in Sinatra. Users won’t wait 15 seconds for an image to be generated, especially if the file is being provided by a server without any loading indication on the page.

Thinking of a Solution

When thinking of optimization ideas, two options stood out to me from Willem’s README:

Can I adapt Virgo to use OilyPNG instead of ChunkyPNG?
Is there another library that implements array manipulation in C?

After some research into OilyPNG, I found that the functions I used with ChunkyPNG weren’t implemented in OilyPNG¹, so I was left to find another library that enabled a faster manipulation of integer arrays. I knew that in the Python world, NumPy would be the immediate answer. After some research for a Ruby alternative to NumPy, I came across NArray, which appeared to be a solution others had relied on in the past. I alluded to this possible approach in my original post:

ChunkyPNG was used to manage and save the wallpaper images, and Commander enabled terminal argument parsing and help documentation. For optimization, I plan to use NArray for creating the pixel locations in the wallpaper and writing a custom PNG encoder.

Fast Pixel Placement

After some profiling, the section of code that could be improved the most was overlaying pixels on the background image:

def place_pixel
  # Create a new canvas
  pixel = Image.new(@pixel_diameter, @pixel_diameter, @theme.foreground)

  # Replace the old image with the new canvas at the pixel coordinate
  @image = @image.replace(pixel, pixel_coordinate.x, pixel_coordinate.y)
end

Rather than creating new ChunkyPNG::Images and overlaying them on a master Image, I decided to use a new data structure for the image. Instead, Virgo now uses an NArray to represent the image, where each item is the Integer representation of the pixel ChunkyPNG::Color. For every pixel, a portion of the array is replaced with the integer color:

def create_map
  # Start with each pixel in the image as the background color
  map = NArray.int(@width, @height).fill!(@theme.background)

  # Place each pixel in the map
  count = number_pixels_to_place
  (1..count).each do
    # Determine pixel location
    x = @x_distribution.random_point
    x_max = x + @pixel_diameter
    y = @y_distribution.random_point
    y_max = y + @pixel_diameter

    # Replace the pixel in the map with a new (random) color
    map[x..x_max, y..y_max] = @theme.random_foreground
  end
end

Then, to create save a Wallpaper instance, a ChunkyPNG::Image is created by inserting rows of the NArray into the Image:

def image
  img = Image.new(@width, @height, Color::TRANSPARENT)

  # Put each row of @map into the image
  num_rows = @map.shape[1]
  (0...num_rows).each do |row_idx|
    # Replace the row in the image with the new colors
    img.replace_row!(row_idx, @map[true, row_idx])
  end

  img
end

`int` vs `Integer`

There was only one problem with this solution: the range of values for ChunkyPNG::Image would often exceed the range of the int type used by NArray. Therefore, most of the colors in the predefined themes could not be placed into the color map as-is.

I decided to implement a color hash (enum) for a Theme instance, where every key is a unique (low Integer value identifier), and each value is the (large Integer) ChunkyPNG::Image value. The background color (@background) will always have the identifier 0, and the foreground colors (@foregrounds) have identifiers from 1 to n. The color hash is created in Theme.initialize:

def initialize(background = BACKGROUNDS[:black],
               foregrounds = FOREGROUNDS[:ruby])
  @background = Color.from_hex(background)
  @foregrounds = foregrounds.map { |x| Color.from_hex(x) }

  # Because NArray can't handle the size of some ChunkyPNG::Color
  # values, create a Hash of the background and foreground colors,
  # where the key of the hash is an Integer, and the
  # value is the value of the color
  # Background has a key of 0, foregrounds have keys from 1..n
  colors = [@background] + @foregrounds
  colors_with_indices = colors.each_with_index.map do |color, idx|
    [idx, color]
  end
  @color_hash = Hash[colors_with_indices]
end

Let’s look at an example:

2.6.3 :001 > # Construct a theme using predefined color names
2.6.3 :002 > t = Theme.from_syms(:black, :ruby)
 => #<Theme:0x00007f861f8c2128
      @background=255,
      @foregrounds=[
        2367954943,
        2720605439,
        3073321215,
        3425971711,
        3561307903,
        3646510079,
        3731712255],
      @color_hash={
        0=>255,
        1=>2367954943,
        2=>2720605439,
        3=>3073321215,
        4=>3425971711,
        5=>3561307903,
        6=>3646510079,
        7=>3731712255}>

Note that the @background color 255 is in @color_hash as 0=>255, which means it has an identifier of 0. The second color in @foregrounds, 2720605439, is in @color_hash as 2=>2720605439, meaning that color has an identifier of 2.

Therefore, if the Wallpaper map has the following values:

[[0, 0, 0],
  [0, 2, 0],
  [0, 0, 0] ]

Then the middle pixel is red (a value of 2720605439), and the border pixels are black (a value of 255).

Next, we add a few helper functions to Theme for retrieving a random foreground (pixel) color and color keys/values:

# Returns the key for the background color
def background_key
  # The background always has a key of 0
  0
end

# Returns a random @color_hash foreground key
def random_foreground_key
  # (Slightly) speed up getting a foreground by returning the first
  # item if only one exists
  color = if @foregrounds.length == 1
            @foregrounds[0]
          else
            @foregrounds.sample
          end

  key_from_color(color)
end

# Returns the ChunkyPNG::Color value for a color key
def color_from_key(key)
  @color_hash[key]
end

# Returns the key (in @color_hash) for a color value
def key_from_color(color)
  @color_hash.key(color)
end

And create_map is updated to use the color keys rather than the large values:

def create_map
  # Start with each pixel in the image as the background color
  map = NArray.int(@width, @height).fill!(@theme.background_key)

  # Place each pixel in the map
  count = number_pixels_to_place
  (1..count).each do
    # Determine pixel location
    x = @x_distribution.random_point
    x_max = x + @pixel_diameter
    y = @y_distribution.random_point
    y_max = y + @pixel_diameter

    # Replace the pixel in the map with a new (random) color
    map[x..x_max, y..y_max] = @theme.random_foreground_key
  end

  map
end

Now we can test whether using NArray vs ChunkyPNG::Image reduced the time to generate large wallpapers.

Results

Using NArray significantly improved Virgo’s speed:

Note the exponential time complexity of the original implementation, and the linear time complexity.² With 10 trials, NArray Virgo took 0.5 seconds on average to generate a 1000x1000 wallpaper, while the original implementation of Virgo takes 15.1 seconds on average. That’s a a 30x improvement! Even better, these updates will allow me to make a responsive Sinatra web frontend, without worrying about user retention or delays.

Future Improvements

I believe more improvements could be made to Virgo, especially since I plan on writing a web frontend. In particular, I’ve added a Distribution class to enable both normal and uniform pixel distributions, but I have not added it as a feature to the CLI. Further, there is a new version of NArray called Numo::NArray that supports UInt64 values, so there will no longer be a need to map each color in a Theme to unique identifiers.

Virgo is available on my GitHub profile, and it’s open to pull requests!

In fact, the library didn’t seem to have implemented much, if any, of the ChunkyPNG functionality. ↩

Well, O(n) at least comparative to the ChunkyPNG implementation and tests I conducted up to an image size of 5000x5000. ↩

Fast Introduction to Node APIs

Nicky Marino — Sat, 23 Nov 2019 16:18:38 +0000

This post was originally published on my website. Check it out for more awesome content!

By the end of this post, we will have created an API using Node, express and body-parser. Our API will have two endpoints: /magic-8-ball will return a random Magic 8-Ball response, and /to-zalgo will convert given text into Zalgo text.

Setup

First, create a new folder named node-api and navigate to it. We need to create a new npm package that will hold our API app. Run the following command and fill out the information. Each part can be left the default, except the entry point should be app.js:

$ npm init

Next, let's install express and body-parser, as we'll need both later:

$ npm install express body-parser

In order to run our app, we'll add a command inside package.json for npm start. Add this item to the "scripts" array:

  "scripts": {
    ...
    "start": "node app.js"
  },

Express Hello World

Now that we have our package set up, we can begin writing the web app. Let's return "Hello world!" at the root of our app (/, or http://localhost:3200/):

// Load the modules we installed
const express = require('express')
const bodyparser = require('body-parser')

// Tell express to run the webserver on port 3200
const app = express();
const port = process.env.port || 3200

// Use body-parser for unencoding API request bodies - more on this later
app.use(bodyparser.json())
app.use(bodyparser.urlencoded({ extended: false }))

app.listen(port, () => {
    console.log(`running on port ${port}`)
})

// Return "Hello world" when you go to http://localhost:3200
app.get('/', (req, res) => res.send('Hello world!'))

To test our app, run npm start in one terminal window, then use curl in the other:

$ curl http://localhost:3200
Hello world!

Magic 8-Ball Responses

Our first API endpoint, /magic-8-ball, will return a JSON result in the form of {"prediction": "<8-ball response>"}. I wrote a helper function to return a random item from an array:

function randomItemFromArray(arr) {
    return arr[Math.floor(Math.random() * arr.length)]
}

Then all we need to do is have our server keep an array of possible responses, pick a random one, and return the response in JSON format:

// Return a random response for http://localhost:3200/magic-8-ball
// {"prediction": "<random_prediction>"}
app.get('/magic-8-ball', (req, res) => {
    const responses = [
        'It is certain.',
        'It is decidedly so.',
        'Without a doubt.',
        'Yes - definitely.',
        'You may rely on it.',
        'As I see it, yes.',
        'Most likely.',
        'Outlook good.',
        'Yes.',
        'Signs point to yes.',
        'Reply hazy, try again.',
        'Ask again later.',
        'Better not tell you now.',
        'Cannot predict now.',
        'Concentrate and ask again.',
        "Don't count on it.",
        'My reply is no.',
        'My sources say no.',
        'Outlook not so good.',
        'Very doubtful.'
    ]

    res.status(200).json({
        prediction: randomItemFromArray(responses)
    })
})

Run npm start, and we can test it a few times using curl:

$ curl http://localhost:3200/magic-8-ball
{"prediction":"Without a doubt."}

$ curl http://localhost:3200/magic-8-ball
{"prediction":"Yes - definitely."}

$ curl http://localhost:3200/magic-8-ball
{"prediction":"Signs point to yes."}

Zalgo Text

Our Zalgo endpoint (/to-zalgo) is a little more advanced. A user will send a POST request including a message in the form {"text": "your text here"}, and the endpoint will return a response in the form {"code": 200, "original": "your text here", "zalgo": "zalgo-ified text"}. The endpoint will also return a 400 HTTP Status Code error if the input data is incorrect:

// Return Zalgo-ified text for http://localhost:3200/to-zalgo
// Input:   {"text": "your text here"}
// Returns: {"code": 200, "original": "your text here", "zalgo": "zalgo-ified text"}
app.post('/to-zalgo', (req, res) => {
    // Return 400 if the input doesn't contain a "text" element
    if (req.body.text === undefined) {
        res.status(400).json({
            "code": 400,
            "message": "Missing 'text' argument"
        })
        return
    }

    original = req.body.text
    zalgo = toZalgo(original)

    res.status(200).json({
        "code": 200,
        "original": original,
        "zalgo": zalgo
    })
})

Let's test it a few times with curl. To send data in a POST request, like our text in JSON format, use -d "data". Because we're sending data in a JSON format, our requests via curl will need to include -H "Content-Type: application/json" as well.

(If you're wondering why the text looks odd, I'd recommend checking out another Zalgo converter first)

$ curl -d '{"text":"Sphinx of black quartz, judge my vow"}' -H "Content-Type: application/json" http://localhost:3200/to-zalgo
{"code":200,"original":"Sphinx of black quartz, judge my vow","zalgo":"S̡̲̳͔̻ͤ̏ͦ̾̀͒̀p̰̯̐̃͒͂ͪͨͤ͘͠h̷̖̰̩̍ͯi̸̩̜͇̪͍͉̭ͨ͐̆͞ͅn̡̧̤̭͚̤̯̼̹ͪͫ́̌ͫ̇̑̋ͅx̧̻̤̄ͩ͋ͣ͂ ̥̤̩̳̠͖ͧ͡ͅö͍̮̅ͯ̋ͣf̠͎̗͕̯̈́̀͑̐͌͊̍͒́ͅ ̦̬̱͉̫͍̞ͤͯͦ͂͜b̡̼̱̊ͅl̵̻̹͇̘̒̌̊̄aͩ̏͛̋̇̅̇ͩ̀͏̘̳̲̫͕ͅc̢̛̗̱͗́̓̆̌k̡͉͉̼̾̍̒͌̀ ̡̳͈͓̞̦̞̱̥̒̌ͦ̅̃q̰̪̟̥̿̀͝ȕ̗a͓̟͍͐̓̂ͣ̀͜r̞̭̪̦̩̹̂̒̐͗̕t̺͎͛̿̽͒̑̓̆ͧz̸͖̟͓̪̻͓̝̦ͨ̕,̻͔͙̲̓̈ͮ̍ ͍̘̟̖̩͊̀̈́ͩͯ̑j̷͕̱̖̔ͧ͌u̗̱͈ͨ̄ͩͬd̜̖̖̦̺̟́̇͐͛̒̆͊ͦ͜g̎ͩͅe̪̟̦͓̥̘͙ͭ̊ͨ̓ ͔̳̒̔̈̈́̈͠ͅm̧̪̊̌y̧͂̑ͯͤ͏͔͔͓͕̮ ̸̛͎͚͇͔̭̱̱͐ͮ̐ͪ͐̊͌v̘̬̘͋̅̽̅̄̐̀o̵̤̝̯̞̪̍͞ͅw̶̠̝̦̹͔̍ͪ͐̂ͮͭ̌͟"}

{"code":200,"original":"the blood of the ancients resides within me","zalgo":"t͍̗͖͚͙͖͖̿ͪ̍h͍̘̩̤̼̞̫̜̒͟ȩ̛̺̫̖̝̰̥͋͛̎̎̈̈ ̢̼̫͈͓ͦ̿ͯb̺̖͚̤͓̲͓ͬ͊ͬ͑̅l̼̪̞̮͖̩̥͕̎ͧ̓̋̐̒ͧͯö̱̹͔̫͇́͌ͭͩ̉̆ͬ͆͠ͅô̸̶̲̫̞͔̻̝̰͓͋d̹̫̠͚͉͎ͨ͑ͯ̀ ̨̫͍̹̺̰̑͛̂̾͗ͪ̓ͅô͙̰͍͓̯͍̼̟ͭ́̽̑́͐̓f̯̥͙͈̺̮̙̙̅̌͂̓ͦ ̸͚̝̥̮̅̾t̨̟̗̟̼͔̑ͥ̊̾ͧͮ̿̿h̜̉͋ͮ͐e̪̳ͧ̾̏ ͬͤ̄̽̾̈̓͊͏̖̗̪͖͚a̢̩̖̯̹͗̊̽͢n̴̔ͥ̓͐͏̙̞̙̭̞͉c̖͕̘̗͉̠̬͂ͤͦ͋ì͕̥̱͍̗̐̅̆̓ͫe̮̩̩̮̬͕͈̾͂͒ͪ͛̇͞n̸̳̹̗͊ͦ̋ͅt͎̯̖̟̫ͯͪs͔̮͋ͧͩ͋̏ͯ̆͢ ̺̤̘̫̗̻̂r̡͚̮͇̘̻͔̉ͅĕ͔̪͖͓̯̙͙͗̂ͯ͛ͭs̵̝̘̺̠̘ͬͮi̴͖̤̟̭͚̞ͪͣd̶̛̪͈̉e͉̺̖̫ͥ̔̽̂̄͒́ͬ́́ͅṡ̵͕͟ͅ ̷̜̤̝̹̦̼͖̅ͭ̈͌͐̍ͦ͗ͅw̧̠͍̻̜͆̔ͣ͗͜i̵̶̙͉̺̦̲̅͋t̗̽͑͐ͣ̇ͣ͛ͧh̢̗͍͎̪̪̹̳̎͗̑̔̎̏͛͜i̶̱̪̺̖̻͓ͥ̿ͨ̇̅̔͗̎ͅņ̪ͬ̇ͭ̉ͬͩ͢ ̶̨̲̩̙ͦ̔̈́̄m̡̳̬̟͐e̱̩̠̙ͨ̓̇̽͑̋"}

Conclusion

Now our API has two endpoints, /magic-8-ball and /to-zalgo for you to use as a starting point for your own web app!

Here's the full version of our app.js:

// Load the modules we installed
const express = require('express')
const bodyparser = require('body-parser')
var toZalgo = require('to-zalgo')

// Tell express to run the webserver on port 3200
const app = express();
const port = process.env.port || 3200

// Use body-parser for unencoding API request bodies - more on this later
app.use(bodyparser.json())
app.use(bodyparser.urlencoded({ extended: false }))

app.listen(port, () => {
    console.log(`running on port ${port}`)
})

// Return "Hello world" when you go to http://localhost:3200
app.get('/', (req, res) => res.send('Hello world!'))

// Return a random response for http://localhost:3200/magic-8-ball
// Returns: {"prediction": "<random_prediction>"}
app.get('/magic-8-ball', (req, res) => {
    const responses = [
        'It is certain.',
        'It is decidedly so.',
        'Without a doubt.',
        'Yes - definitely.',
        'You may rely on it.',
        'As I see it, yes.',
        'Most likely.',
        'Outlook good.',
        'Yes.',
        'Signs point to yes.',
        'Reply hazy, try again.',
        'Ask again later.',
        'Better not tell you now.',
        'Cannot predict now.',
        'Concentrate and ask again.',
        "Don't count on it.",
        'My reply is no.',
        'My sources say no.',
        'Outlook not so good.',
        'Very doubtful.'
    ]

    res.status(200).json({
        prediction: randomItemFromArray(responses)
    })
})

// Return Zalgo-ified text for http://localhost:3200/to-zalgo
// Input:   {"text": "your text here"}
// Returns: {"code": 200, "original": "your text here", "zalgo": "zalgo-ified text"}
app.post('/to-zalgo', (req, res) => {
    // Return 400 if the input doesn't contain a "text" element
    if (req.body.text === undefined) {
        res.status(400).json({
            "code": 400,
            "message": "Missing 'text' argument"
        })
        return
    }

    original = req.body.text
    zalgo = toZalgo(original)

    res.status(200).json({
        "code": 200,
        "original": original,
        "zalgo": zalgo
    })
})

function randomItemFromArray(arr) {
    return arr[Math.floor(Math.random() * arr.length)]
}

The entire example app can be found as a GitHub repo as well.

Using Levenshtein Distances to Find Similar Strings

Nicky Marino — Wed, 23 Oct 2019 18:29:58 +0000

This post was originally published on my website. Check it out for more awesome content!

Imagine you're writing a mobile app, and your user searches for the word kitten. Unfortunately, the only search terms you expected them to enter were from the following:

smitten
mitten
kitty
fitting
written

How do we figure out which word the user meant to type?

Levenshtein Distances

A Levenshtein distance is a distance between two sequences a and b. If a and b are strings, the Levenshtein distance is the minimum amount of character edits needed to change one of the strings into the other. There are three types of edits allowed:

Insertion: a character is added to a
Deletion: a character is removed from b
Substitution: a character is replaced in a or b

For example, if the first string a = 'abc' and the second string is b = 'abc', the Levenshtein distance between the two strings is 0 because a and b are equal. If a = 'abcd' and b = 'a', the distance is 3. If a = 'abcd' and b = 'aacc', the distance is 2.

The definition of the Levenshtein distance for a string a with a length i and a string b with a length j is:

This definition is a recursive function. The first portion, max(i, j) if min(i, j) = 0, is the base cases where either the first string or the second string is empty.

The function 1_(ai != bi) at the end of the third minimum element is the cost. If a[i] != b[i], the cost is 1, otherwise the cost is 0. The first minimum element is a deletion from a, the second is an insertion, and the third is a substitution.

A Naive Implementation

First, let's implement a straightforward implementation in Swift. We'll create a function named levenshtein_distance and write the base cases to check whether either of the strings are empty:

func levenshtein_distance(a: String, b: String) -> Int {
    // If either array is empty, return the length of the other array
    if (a.count == 0) {
        return b.count
    }
    if (b.count == 0) {
        return a.count
    }
}

Then we add the recursive portion. We calculate the cost for the substitution, then find the minimum distance between the three different possible edits (deletion, insertion, or substitution):

func levenshtein_distance(a: String, b: String) -> Int {
    // ...

    // Check whether the last items are the same before testing the other items
    let cost = (a.last == b.last) ? 0 : 1

    let a_dropped = String(a.dropLast())
    let b_dropped = String(b.dropLast())

    return min(
        // Find the distance if an item in a is removed
        levenshtein_distance(a: a_dropped, b: b) + 1,
        // Find the distance if an item is removed from b (i.e. added to a)
        levenshtein_distance(a: a, b: b_dropped) + 1,
        // Find the distance if an item is removed from a and b (i.e. substituted)
        levenshtein_distance(a: a_dropped, b: b_dropped) + cost
    )
}

Let's test our distance function with a simple test case:

print(opti_leven_distance(a: "123", b: "12"))

More example test cases can be found below in the final files. And now we can compare the distances of our words to the string kitten to figure out which word the user probably meant to type:

// Print out the distances for our test case
let first_word = "kitten"
let test_words = ["smitten", "mitten", "kitty", "fitting", "written"]

for word in test_words {
    let dist = opti_leven_distance(a: first_word, b: word)
    print("Distance between \(first_word) and \(word): \(dist)")
}

Distance between kitten and smitten: 2
Distance between kitten and mitten: 1
Distance between kitten and kitty: 2
Distance between kitten and fitting: 3
Distance between kitten and written: 2

The user probably meant to type mitten instead of kitten!

An Improved Implementation

The recursive implementation of the Levenshtein distance above won't scale very well for larger strings. What if we needed to find the distance between a thousand strings, each with hundreds of characters?

One improved way to calculate a Levenshtein distance is to use a matrix of distances to "remember" previously calculated distances. First, the distance function should check for empty strings. Then, we'll create a matrix to hold the distance calculations:

func opti_leven_distance(a: String, b: String) -> Int {
    // Check for empty strings first
    if (a.count == 0) {
        return b.count
    }
    if (b.count == 0) {
        return a.count
    }

    // Create an empty distance matrix with dimensions len(a)+1 x len(b)+1
    var dists = Array(repeating: Array(repeating: 0, count: b.count+1), count: a.count+1)
}

The first column and first row of the distance matrix are zeros as an initialization step. The next column goes from 1 to the length of a to represent removing each character to get to an empty string, and the next row goes from 1 to the length of b to represent adding (or inserting) each character to get to the value of b:

func opti_leven_distance(a: String, b: String) -> Int {
    //...

    // a's default distances are calculated by removing each character
    for i in 1...(a.count) {
        dists[i][0] = i
    }
    // b's default distances are calulated by adding each character
    for j in 1...(b.count) {
        dists[0][j] = j
    }
}

Similar to our naive implementation, we'll check the remaining indices in the distance matrix. This time, however, we'll use the previous values stored in the matrix to calculate the minimum distance rather than recursively calling the distance function. The final distance is the last element in the distance matrix (at the bottom right):

func opti_leven_distance(a: String, b: String) -> Int {
    //...

    // Find the remaining distances using previous distances
    for i in 1...(a.count) {
        for j in 1...(b.count) {
            // Calculate the substitution cost
            let cost = (a[i-1] == b[j-1]) ? 0 : 1

            dists[i][j] = min(
                // Removing a character from a
                dists[i-1][j] + 1,
                // Adding a character to b
                dists[i][j-1] + 1,
                // Substituting a character from a to b
                dists[i-1][j-1] + cost
            )
        }
    }

    return dists.last!.last!
}

We can use our test cases again to verify that our improved implementation is correct:

print(opti_leven_distance(a: "123", b: "12"))

// Print out the distances for our test case
let first_word = "kitten"
let test_words = ["smitten", "mitten", "kitty", "fitting", "written"]

for word in test_words {
    let dist = opti_leven_distance(a: first_word, b: word)
    print("Distance between \(first_word) and \(word): \(dist)")
}

1
Distance between kitten and smitten: 2
Distance between kitten and mitten: 1
Distance between kitten and kitty: 2
Distance between kitten and fitting: 3
Distance between kitten and written: 2

Swift and Python Implementations

Distance.playground:


import Foundation


/// Calculates the Levenshtein distance between two strings
/// - Parameter a: The first string
/// - Parameter b: The second string
func levenshtein_distance(a: String, b: String) -> Int {
    // If either array is empty, return the length of the other array
    if (a.count == 0) {
        return b.count
    }
    if (b.count == 0) {
        return a.count
    }

    // Check whether the last items are the same before testing the other items
    let cost = (a.last == b.last) ? 0 : 1

    let a_dropped = String(a.dropLast())
    let b_dropped = String(b.dropLast())

    return min(
        // Find the distance if an item in a is removed
        levenshtein_distance(a: a_dropped, b: b) + 1,
        // Find the distance if an item is removed from b (i.e. added to a)
        levenshtein_distance(a: a, b: b_dropped) + 1,
        // Find the distance if an item is removed from a and b (i.e. substituted)
        levenshtein_distance(a: a_dropped, b: b_dropped) + cost
    )
}

/// String extension to add substring by Int (such as a[i-1])
extension String {
    subscript (i: Int) -> Character {
      return self[index(startIndex, offsetBy: i)]
    }
}

/// A more optimized version of the Levenshtein distance function using an array of previously calculated distances
/// - Parameter a: The first string
/// - Parameter b: The second string
func opti_leven_distance(a: String, b: String) -> Int {
    // Check for empty strings first
    if (a.count == 0) {
        return b.count
    }
    if (b.count == 0) {
        return a.count
    }

    // Create an empty distance matrix with dimensions len(a)+1 x len(b)+1
    var dists = Array(repeating: Array(repeating: 0, count: b.count+1), count: a.count+1)

    // a's default distances are calculated by removing each character
    for i in 1...(a.count) {
        dists[i][0] = i
    }
    // b's default distances are calulated by adding each character
    for j in 1...(b.count) {
        dists[0][j] = j
    }

    // Find the remaining distances using previous distances
    for i in 1...(a.count) {
        for j in 1...(b.count) {
            // Calculate the substitution cost
            let cost = (a[i-1] == b[j-1]) ? 0 : 1

            dists[i][j] = min(
                // Removing a character from a
                dists[i-1][j] + 1,
                // Adding a character to b
                dists[i][j-1] + 1,
                // Substituting a character from a to b
                dists[i-1][j-1] + cost
            )
        }
    }

    return dists.last!.last!
}

/// Function to test whether the distance function is working correctly
/// - Parameter a: The first test string
/// - Parameter b: The second test string
/// - Parameter answer: The expected answer to be returned by the distance function
func test_distance(a: String, b: String, answer: Int) -> Bool {
    let d = opti_leven_distance(a: a, b: b)

    if (d != answer) {
        print("a: \(a)")
        print("b: \(b)")
        print("expected: \(answer)")
        print("distance: \(d)")
        return false
    } else {
        return true
    }
}

// Test the distance function with many different examples
test_distance(a: "", b: "", answer: 0)
test_distance(a: "1", b: "1", answer: 0)
test_distance(a: "1", b: "2", answer: 1)
test_distance(a: "12", b: "12", answer: 0)
test_distance(a: "123", b: "12", answer: 1)
test_distance(a: "1234", b: "1", answer: 3)
test_distance(a: "1234", b: "1233", answer: 1)
test_distance(a: "1248", b: "1349", answer: 2)
test_distance(a: "", b: "12345", answer: 5)
test_distance(a: "5677", b: "1234", answer: 4)
test_distance(a: "123456", b: "12345", answer: 1)
test_distance(a: "13579", b: "12345", answer: 4)
test_distance(a: "123", b: "", answer: 3)
test_distance(a: "kitten", b: "mittens", answer: 2)

print(opti_leven_distance(a: "123", b: "12"))

// Print out the distances for our test case
let first_word = "kitten"
let test_words = ["smitten", "mitten", "kitty", "fitting", "written"]

for word in test_words {
    let dist = opti_leven_distance(a: first_word, b: word)
    print("Distance between \(first_word) and \(word): \(dist)")
}

Here's a Python implementation of the Swift code above as distance.py. The Python version also can handle any list as well as any str

# Calculates the Levenshtein distance between two strings
def levenshtein_distance(a, b):
    # If either array is empty, return the length of the other array
    if not len(a):
        return len(b)
    if not len(b):
        return len(a)

    # Check whether the last items are the same before testing the other items
    if a[-1] == b[-1]:
        cost = 0
    else:
        cost = 1

    return min(
        # Find the distance if an item in a is removed
        levenshtein_distance(a[:-1], b) + 1,
        # Find the distance if an item is removed from b (i.e. added to a)
        levenshtein_distance(a, b[:-1]) + 1,
        # Find the distance if an item is removed from a and b (i.e. substituted)
        levenshtein_distance(a[:-1], b[:-1]) + cost
    )

# A more optimized version of the Levenshtein distance function using an array of previously calculated distances
def opti_leven_distance(a, b):
    # Create an empty distance matrix with dimensions len(a)+1 x len(b)+1
    dists = [ [0 for _ in range(len(b)+1)] for _ in range(len(a)+1) ]

    # a's default distances are calculated by removing each character
    for i in range(1, len(a)+1):
        dists[i][0] = i
    # b's default distances are calulated by adding each character
    for j in range(1, len(b)+1):
        dists[0][j] = j

    # Find the remaining distances using previous distances
    for i in range(1, len(a)+1):
        for j in range(1, len(b)+1):
            # Calculate the substitution cost
            if a[i-1] == b[j-1]:
                cost = 0
            else:
                cost = 1

            dists[i][j] = min(
                # Removing a character from a
                dists[i-1][j] + 1,
                # Adding a character to b
                dists[i][j-1] + 1,
                # Substituting a character from a to b
                dists[i-1][j-1] + cost
            )

    return dists[-1][-1]

# Function to test whether the distance function is working correctly
def test_distance(a, b, answer):
    dist = opti_leven_distance(a, b)

    if dist != answer:
        print('a:', a)
        print('b:', b)
        print('expected:', answer)
        print('distance:', dist)
        print()

if __name__ == '__main__':
    # Test the distance function with many different examples
    test_distance('', '', 0)
    test_distance('1', '1', 0)
    test_distance('1', '2', 1)
    test_distance('12', '12', 0)
    test_distance('123', '12', 1)
    test_distance('1234', '1', 3)
    test_distance('1234', '1233', 1)
    test_distance([1, 2, 4, 8], [1, 3, 4, 16], 2)
    test_distance('', '12345', 5)
    test_distance([5, 6, 7, 7], [1, 2, 3, 4], 4)
    test_distance([1, 2, 3, 4, 5, 6], [1, 2, 3, 4, 5], 1)
    test_distance([1, 3, 5, 7, 9], [1, 2, 3, 4, 5], 4)
    test_distance([1, 2, 3], [], 3)
    test_distance('kitten', 'mittens', 2)



    first_word = 'kitten'
    test_words = ['smitten', 'mitten', 'kitty', 'fitting', 'written']
    for word in test_words:
        dist = opti_leven_distance(first_word, word)
        print(f'Distance between {first_word} and {word}: {dist}')

DEV Community: Nicky Marino

Use Pyenv to Streamline your Python Projects

Overview

Install Pyenv

Install Python 3.8.2

Manage Pyenv Versions

How to Use Pyenv Virtual Environments

Helpful Commands

Add Your Vimrc to Obsidian

How to Install Ruby 2.7.3 on M1 Mac

Why 2.7.3?

The Solution

The Issue

Flags

Rosetta

Usename Macro Error

Next Steps

A Beginner's Guide to Flask and Replit

Creating a Flask Project

Hello, World!

HTML Assets and CSS Styles

Adding an Index Page

Adding CSS Styling

How to Draft Blog Posts in Notion

Markdown, Jekyll, and Drafts! Oh My!

The Notion Database

Creating the Table

Table Properties

The Page is as Mighty as the Table

Hello World

You are Your Own Publisher

Great, Now There’s Two of Them

How to Write List Comprehensions with Python

For Loops

List Comprehension

List Comprehension Syntax

Examples

Conditionals

Key Points

Python's Walrus Operator

Named Expressions

Benefits

Don’t Repeat Yourself

Reuse Variables

Legible Code Blocks

Wrap Up

Multiple Named Profiles with AWS CLI

Adding a Named Profile

Setting a Profile for Kubeconfig

How to Use Jekyll on macOS Catalina with RVM

Installing RVM and Ruby

Installing Jekyll

Optimizing a Ruby Script with NArray

Improvement Goal

Thinking of a Solution

Fast Pixel Placement

int vs Integer

Results

Future Improvements

Fast Introduction to Node APIs

Setup

Express Hello World

Magic 8-Ball Responses

Zalgo Text

Conclusion

Using Levenshtein Distances to Find Similar Strings

Levenshtein Distances

A Naive Implementation

An Improved Implementation

Swift and Python Implementations

`int` vs `Integer`