DEV Community: Sam Agnew

How to Write Lua Scripts for Video Games with the BizHawk Emulator

Sam Agnew — Fri, 31 Jan 2025 22:16:30 +0000

BizHawk is a multi-system emulator beloved by the Tool Assisted Speedrun community for its recording/playback and debugging tools, as well as Lua scripting functionality that can be used for a variety of purposes.

While there is basic documentation describing some of the functions available in these scripts, the lack of working code samples might make it difficult for some to get started. Let's walk through some of the Lua scripting features the BizHawk emulator provides, and have some fun with real examples.

Setting up the BizHawk emulator

BizHawk runs on multiple operating systems, but Lua scripting is only available on the Windows versions. If you are on Mac or Linux, you can use Wine, although setup might be a bit of an involved process depending on which version of which operating system you're using.

There are installation instructions in the README of the project's repository, including an installer that takes care of the prerequisites. Download, unzip, run this installation tool, and then download the corresponding version of the emulator from TASVideos. Now you can run the EmuHawk executable, and load up your favorite video game ROM!

The original Super Mario Bros for the Nintendo Entertainment System is a widely popular classic game so we will be referencing it throughout the post for our code examples.

Getting started with Lua scripting

Now that you have the emulator running, open up the Lua scripting window via Tools -> Lua Console. Here you can open scripts, control their execution, and view their output. In the text editor of your choice, create a file called hello.lua and add the following line of code to it:

console.log('Hello World!')

Run this code by clicking Script -> Open Script in the Lua Console and navigating to where you saved the file. You should see "Hello World!" printed in the “Outputs” tab.

For most emulator scripts, it's desirable to have a main execution loop that runs continuously until it's stopped, executing code before each frame of the game is rendered. There are two ways you can do this. One is to write an infinite loop using emu.frameadvance at the end of each iteration, and the other is to register a function to event.onframestart. Let's take a look at both of these.

Replace the code in hello.lua with the following, which will write "Hello World!" to the screen for every frame of the game, rather than the console:

while true do
    gui.text(50, 50, 'Hello World!')
    emu.frameadvance()
end

You can double click your script in the Lua console to toggle it on and off, and each time you restart it, your new code will load if you have made changes and saved the file. Do this and you should see "Hello World!" pop onto the screen and stay there because it is being printed on every frame. Personally, I am excited to write an infinite loop that's actually useful, as an act of vindication for all of the ones I've written by mistake.

If you don't like the way infinite loops look, let's try doing it more functionally using event.onframestart with the following code that has the same behavior:

With that out of the way, let's move onto writing code to interact with the game itself.

Reading from and writing to memory

BizHawk provides many useful developer tools to gain insight into the games you are playing. These utilities come in handy when doing speedruns or hacking old games. One of them is a “hex editor”, a tool that allows you to view and edit the game's RAM in real time. The RAM is displayed in the form of 4 digit (for NES games) hexadecimal addresses and 2 digit hexadecimal values.

Open BizHawk's hex editor by clicking Tools -> Hex Editor. All of the two digit hex numbers you see represent a value in the game’s RAM at a specific location. You can change any value to see what happens in-game. You can also search for specific values using Ctrl-F or clicking Edit -> Find. For example, the hex editor can be used to find the bytes corresponding to the timer in Super Mario Bros and to change the time to zero to kill Mario:

Cleverly editing memory addresses can result in some interesting gameplay modifications. In this case, we killed Mario by editing the values in the memory addresses, 07F8-07FA to zero. But this can also be done programmatically in a Lua script.

Create a new script with the following code, which will log the values of the timer to the console, and then change them to zero to kill Mario:

console.log(memory.readbyte(0x07F8) .. memory.readbyte(0x07F9) .. memory.readbyte(0x07FA))

memory.writebyte(0x07F8, 0)
memory.writebyte(0x07F9, 0)
memory.writebyte(0x07FA, 0)

In Lua, 0x before a number means that you are referring to a hexadecimal value, and .. is an operator for string concatenation. As you can see in this code you can read the values of hex addresses in the game's memory with memory.readbyte and write to them with memory.writebyte.

The idea of digging around a game's memory may seem daunting at first, but it turns out that for many games, people on the internet have already done this for you! Here is a RAM map for Super Mario Bros. Try playing around with these memory addresses and values for yourself and see what happens.

Controller input - playing the game with code

Aside from manipulating a game's memory, you can also programmatically enter button presses with joypad.set and read input with joypad.get. The button names are different depending on each video game console, so here is a useful chart with string values of button names for each system.

Controller inputs are represented by a table in Lua, which is similar to a dictionary in Python or an object in JavaScript. For instance, joypad.set({A=true}, 1) would press the A button on player 1's controller.

Using Mario as an example again, run a Lua script with the following code to make Mario run to the right indefinitely at full speed:

while true do
    joypad.set({Right=true, B=true}, 1)
    emu.frameadvance()
end

This is great and while running full speed into a Goomba might be funny, it is not a good recipe for success. What if we combined all of the things we learned to write some code that can run through the entire first level of Super Mario Bros? With the information from the RAM map for the game, we can write code to read values from the game's memory to make decisions on when to jump.

Create a file called level1.lua with the following code:

-- A table of indexes of pit locations in the level, as well as
-- two other spots that aren't pits where jumping also helps.
local pits = {[0x1C]=true, [0x24]=true, [0x30]=true, [0x34]=true, [0x3C]=true, [0x54]=true}

while true do
        -- Our button inputs, which will continuously move to the right by default.
        -- Add to this table to jump depending on the situation.
        local input = {Right=true}

        -- Mario's current position
        local mario = memory.readbyte(0x0086)

        -- The first enemy's (on the screen) current position
        local enemy = memory.readbyte(0x0087)

        -- Where we are in the level
        local level_index = memory.readbyte(0x072C)

        -- If Mario is colliding with an object (like a pipe), hold jump for 20 frames.
        if memory.readbyte(0x0490) == 0xFE then
            input.A = true
            for i=1, 20, 1 do
                joypad.set(input, 1)
                emu.frameadvance()
            end

            -- For some reason the above strategy isn't working in these two specific spots.
            if level_index == 0x52 or level_index == 0x60 then
                joypad.set({A=true})
                emu.frameadvance()
            end
        end

        -- If Mario is close to an enemy, jump.
        -- 38 is just a number of pixels that I found works well from experimentation.
        if math.abs(mario - enemy) < 38 then
            input.A = true
        end

        -- If Mario is near a pit, jump.
        if pits[memory.readbyte(0x072C)] then
            -- For some pits, it works better to slow down a bit before jumping.
            -- Advancing a few frames with no button input will do this.
            for i=1, 3, 1 do
                emu.frameadvance()
            end

            input.A = true

            -- Hold A for 20 frames to clear any gaps.
            for i=1, 20, 1 do
                joypad.set(input, 1)
                emu.frameadvance()
            end
        end

        -- Set the button inputs and advance the frame, restarting the loop.
        joypad.set(input, 1)
        emu.frameadvance()
end

I left comments for each section to explain the code, but the basic strategy is to continuously move to the right while jumping in these scenarios:

Whenever an enemy gets too close
When there is a pit nearby
If Mario is colliding with an obstacle

jumping whenever an enemy gets too close, whenever there is a pit nearby, or whenever Mario is colliding with an obstacle. There are values in memory that represent all of these things, at least accurately enough for this example.

Start the game at World 1-1, and run this code to see Mario complete the level! (If you let it keep running he will likely get to World 1-2 and lose to the second Goomba).

It's not the most sophisticated or efficient way to beat the level, but it gets the job done. This was only the first level, so now the rest is up to you. Try improving on the script to beat the next level!

Level Complete!

We've covered some decent ground here to get you started writing with Lua scripts for video games, but BizHawk's Lua API has some other features as well. With the event module you can register functions to events, such as saving or loading save states, or writing to certain memory locations. There is also a comm module that provides some communications functionality such as socket connections and HTTP requests. These are all described in the BizHawk Lua Functions documentation on TASVideos.

I cannot wait to see what kind emulator extensions you build. Drop me a line if you have any questions or if you just want to show off your hack.

5 Awesome Vim Tricks to Spice up your .vimrc

Sam Agnew — Thu, 23 Jan 2025 20:24:23 +0000

If there's one thing Vim users love more than saving a few seconds per day on keystrokes, it's trading tips and tricks with others about their own setup. As someone who's been using Vim for over 10 years now as his primary text editor, I'm going to do my part by sharing some neat things from my .vimrc that people have asked me about at conferences and hackathons in the past.

For each of these examples, I'll provide the code for you to copy/paste into your own .vimrc and try to explain what's going on. Take what you think works for you and leave the rest!

Crosshair style cursor highlighting

One thing that people ask me about often is the crosshair style cursor that I use in my configuration. All you need to do to make your cursor look like this:

...is add these lines to your .vimrc file (typically located in your home directory, or ~ on most unix based systems):

" Allow crosshair cursor highlighting.
hi CursorLine   cterm=NONE ctermbg=0
hi CursorColumn cterm=NONE ctermbg=0
nnoremap <Leader>c :set cursorline! cursorcolumn!<CR>

This will enable horizontal and vertical highlighting for the row and column you're currently navigating in your file. Honestly, I just think it looks cool. I also included a command that allows you to toggle this on and off, by pressing your leader key followed by c (for "cursor" or "crosshair"). You can change this to fit your liking though.

For the record, I have my leader key set to , and am preserving the original , functionality by remapping with the line nnoremap <leader>, , so that if I ever want to repeat my most recent f or t command in reverse, which I don't use frequently, I just have to hit the key twice.

Copying to your system clipboard

Not being easily able to copy and paste from the system clipboard is a common complaint that many newcomers have when getting started with Vim. While some users can just use the * or + registers to refer to their system clipboard, this doesn't work by default in every installation of Vim. It particularly tends not to work in the Vim that comes pre-installed on many versions of OSX. So here is another way to copy to the system clipboard on OSX:

" Copies the currently selected text to the system clipboard (OSX)
let @c = ':w !pbcopy'

And here is one for pasting:

let @p = ':r !pbpaste'

These are macros that you would add to your .vimrc. To copy or paste with your system's clipboard in Vim with these you would use @c and @p respectively, after selecting text with visual mode (press v).

Alternatively, for some Linux distros, pbcopy can be replaced with xclip -i -sel c or xsel -i -b and pbpaste with xclip -o -sel -c or xsel -o -b.

On another note, if you're using this method it's a good idea to make sure you have a quick way to toggle back and forth between paste mode, for when you are going to be pasting large amounts of text into Vim. I do that with the following line:

set pastetoggle=<leader>p

Alternatively, if you do compile Vim with your equivalent X11 support to be able to use the * or + registers (in Vim type :reg and see if these are in the list of registers to check for availability), then you can change your system's clipboard to be your default clipboard in Vim. I personally don't prefer this method as I actually enjoy having my Vim clipboards be separate from my system clipboard. But you might want to be able to just use the y key to paste whatever you have in your clipboard (like code from Stackoverflow) or c to copy whatever you want from Vim to paste into another program (like your code that you want to post on Stackoverflow) without pressing an extra key. This way, your copying and pasting habits in Vim function as you would expect them to relative to other program. Add the following line to your .vimrc for that behavior:

set clipboard=unnamedplus

Changing whitespace size based on file type

Many polyglot developers, or anyone who writes code in multiple different languages or works on a variety of projects, might want to have different whitespace settings for different file types. I write a lot of Python, so to follow PEP8 style compliance I typically use four-space tabs, but when I switch to writing JavaScript I prefer two-spaced tabs as there tends to be more code nesting.

You can configure Vim to behave differently when it comes to whitespace in different types of files. My default settings when it comes to whitespace are the following:

set tabstop=4 "Sets indent size of tabs"
set softtabstop=4 "Soft tabs"
set expandtab "Turns tabs into spaces"
set shiftwidth=4 "Sets auto-indent size"
set autoindent "Turns on auto-indenting"
set copyindent "Copy the previous indentation on autoindenting"
set smartindent "Remembers previous indent when creating new lines"

But to change those for specific file types, I added the following lines:

au BufNewFile,BufRead *.js,*.jsx*,*.rb,*.html,*.xml :setlocal sw=2 ts=2 sts=2
au FileType make setlocal noexpandtab

You can configure these to your liking depending on what kind of code you tend to work with.

Remapping colon to semicolon, for convenience

This one is fairly simple, yet polarizing. It might seem unimportant, but the : key is probably one of the most frequently used things in Vim. Remapping it to ; prevents you from having to constantly press the shift key. However, this overrides the default functionality of the ; key, which can be used to repeat your most recent f or t commands, something that some people use religiously and others not as often. To account for this, I also add a mapping to place that functionality behind the leader key as I use it far less often:

nnoremap <leader>; ;
map ; :

With this mapping, instead of having to press shift all the time to enter commands, you would be adding an extra keypress to the ; functionality. There already exist discussions about this remapping, with strong opinions on both sides. Your mileage may vary and you might have different preferences or habits than I do, but I personally love just hitting semicolon to enter commands.

Search settings and highlighting

Another thing that anybody who writes code does very often is search for phrases or keywords. It's a good idea to enable some settings to increase the usability of this behavior, such as adding highlighting to your search terms. In my .vimrc, this is what I have for my search settings:

set hlsearch "Highlights search terms"
set incsearch "Highlights search terms as you type them"
set showmatch "Highlights matching parentheses"
set ignorecase "Ignores case when searching"
set smartcase "Unless you put some caps in your search term"

I decided to use the return key because somebody on the internet suggested it, and I don't already use it for anything else. You can change this to fit your preferences. Here's a quick example of what it looks like. The highlighting is going away when I hit the enter key.

:wq!

Hopefully you've found some of these interesting or helpful. There might be stuff in here that you disagree with because they might go against your habits, preferences, or personal Vim philosophy. So in any of those cases, I would suggest taking what you like and leaving the rest. Often, Vim users talk about these things as if there is an objectively "correct" way to do things, but I personally like to think of Vim in terms of a musical instrument. As a Vim enthusiast, I am always looking forward to hearing about more things I can add to my .vimrc. I don't consider myself an expert so there is a whole world of Vim sorcery to discover!

Feel free to contact me about some Vim tricks that you think are cool, ask questions about any of these, or want to express your undying anger at any of my suggestions:

Working with Files Asynchronously in Python using aiofiles and asyncio

Sam Agnew — Tue, 14 Jan 2025 22:21:44 +0000

Asynchronous code has become a mainstay of Python development. With asyncio becoming part of the standard library and many third party packages providing features compatible with it, this paradigm is not going away anytime soon.

If you're writing asynchronous code, it's important to make sure all parts of your code are working together so one aspect of it isn't slowing everything else down. File I/O can be a common blocker on this front, so let's walk through how to use the aiofiles library to work with files asynchronously.

Starting with the basics, this is all the code you need to read the contents of a file asynchronously (within an async function):

async with aiofiles.open('filename', mode='r') as f:
    contents = await f.read()
print(contents)

Let's move on and dig deeper.

What is non-blocking code?

You may hear terms like "asynchronous", "non-blocking" or "concurrent" and be a little confused as to what they all mean. According to this much more detailed tutorial, two of the primary properties are:

Asynchronous routines are able to “pause” while waiting on their ultimate result to let other routines run in the meantime.
Asynchronous code, through the mechanism above, facilitates concurrent execution. To put it differently, asynchronous code gives the look and feel of concurrency.

So asynchronous code is code that can hang while waiting for a result, in order to let other code run in the meantime. It doesn't "block" other code from running so we can call it "non-blocking" code.

The asyncio library provides a variety of tools for Python developers to do this, and aiofiles provides even more specific functionality for working with files.

Setting Up

Make sure to have your Python environment setup before we get started. Follow this guide up through the virtualenv section if you need some help. Getting everything working correctly, especially with respect to virtual environments is important for isolating your dependencies if you have multiple projects running on the same machine. You will need at least Python 3.7 or higher in order to run the code in this post.

Now that your environment is set up, you’re going to need to install some third party libraries. We’re going to use aiofiles so install this with the following command after activating your virtual environment:

pip install aiofiles==0.6.0

For the examples in the rest of this post, we'll be using JSON files of Pokemon API data corresponding to the original 150 Pokemon. You can download a folder with all of those here. With this you should be ready to move on and write some code.

Reading from a file with aiofiles

Let's begin with by simply opening a file corresponding to a particular Pokemon, parsing its JSON into a dictionary, and printing out its name:

import aiofiles
import asyncio
import json


async def main():
    async with aiofiles.open('articuno.json', mode='r') as f:
        contents = await f.read()
    pokemon = json.loads(contents)
    print(pokemon['name'])

asyncio.run(main())

When running this code, you should see "articuno" printed to the terminal. You can also iterate through the file asynchronously, line by line (this code will print out all 9271 lines of articuno.json):

import aiofiles
import asyncio

async def main():
    async with aiofiles.open('articuno.json', mode='r') as f:
        async for line in f:
            print(line)

asyncio.run(main())

Writing to a file with aiofiles

Writing to a file is also similar to standard Python file I/O. Let's say we wanted to create files containing a list of all moves that each Pokemon can learn. For a simple example, here's what we would do for the Pokemon Ditto, who can only learn the move "transform":

import aiofiles
import asyncio

async def main():
    async with aiofiles.open('ditto_moves.txt', mode='w') as f:
        await f.write('transform')

asyncio.run(main())

Let's try this with a Pokemon that has more than one move, like Rhydon:

import aiofiles
import asyncio
import json


async def main():
    # Read the contents of the json file.
    async with aiofiles.open('rhydon.json', mode='r') as f:
        contents = await f.read()

    # Load it into a dictionary and create a list of moves.
    pokemon = json.loads(contents)
    name = pokemon['name']
    moves = [move['move']['name'] for move in pokemon['moves']]

    # Open a new file to write the list of moves into.
    async with aiofiles.open(f'{name}_moves.txt', mode='w') as f:
        await f.write('\n'.join(moves))


asyncio.run(main())

If you open up rhydon_moves.txt you should see a file with 112 lines that starts something like this.

Using asyncio to go through many files asynchronously

Now let's get a little more complicated and do this for all 150 Pokemon that we have JSON files for. Our code will have to read from every file, parse the JSON, and rewrite each Pokemon's moves to a new file:

import aiofiles
import asyncio
import json
from pathlib import Path


directory = 'directory/your/files/are/in'


async def main():
    pathlist = Path(directory).glob('*.json')

    # Iterate through all json files in the directory.
    for path in pathlist:
        # Read the contents of the json file.
        async with aiofiles.open(f'{directory}/{path.name}', mode='r') as f:
            contents = await f.read()

        # Load it into a dictionary and create a list of moves.
        pokemon = json.loads(contents)
        name = pokemon['name']
        moves = [move['move']['name'] for move in pokemon['moves']]

        # Open a new file to write the list of moves into.
        async with aiofiles.open(f'{directory}/{name}_moves.txt', mode='w') as f:
            await f.write('\n'.join(moves))


asyncio.run(main())

After running this code, you should see the directory of Pokemon files populated with .txt files alongside the .json ones, containing move lists corresponding to each Pokemon.

If you need to perform some asynchronous actions and want to end with data corresponding to those asynchronous tasks, such as a list with each Pokemon's moves after having written the files, you can use asyncio.ensure_future and asyncio.gather.

You can break out the portion of your code that handles each file into its own async function, and append promises for those function calls to a list of tasks. Here's an example of what that function, and your new main function would look like:

async def write_pokemon_moves(filename):
    # Read the contents of the json file.
    async with aiofiles.open(f'{directory}/{filename}', mode='r') as f:
        contents = await f.read()

    # Load it into a dictionary and create a list of moves.
    pokemon = json.loads(contents)
    name = pokemon['name']
    moves = [move['move']['name'] for move in pokemon['moves']]

    # Open a new file to write the list of moves into.
    async with aiofiles.open(f'{directory}/{name}_moves.txt', mode='w') as f:
        await f.write('\n'.join(moves))
    return { 'name': name, 'moves': moves }


async def main():
    pathlist = Path(directory).glob('*.json')

    # A list to be populated with async tasks.
    tasks = []

    # Iterate through all json files in the directory.
    for path in pathlist:
        tasks.append(asyncio.ensure_future(write_pokemon_moves(path.name)))

    # Will contain a list of dictionaries containing Pokemons' names and moves
    moves_list = await asyncio.gather(*tasks)

This is a common way to utilize asynchronous code in Python, and is often used for things like making HTTP requests.

So what do I use this for?

The examples in this post using data from the Pokemon were just an excuse to show the functionality of the aiofiles module, and how you would write code to navigate through a directory of files for reading and writing. Hopefully, you can adapt these code samples to the specific problems you're trying to solve so file I/O doesn't become a blocker in your asynchronous code.

We have only scratched the surface of what you can do with aiohttp and asyncio, but I hope that this has made starting your journey into the world of asynchronous Python a little easier.

I’m looking forward to seeing what you build. Feel free to reach out and share your experiences or ask any questions.

Asynchronous HTTP Requests in Python with HTTPX and asyncio

Sam Agnew — Fri, 10 Jan 2025 19:41:41 +0000

Asynchronous code has increasingly become a mainstay of Python development. With asyncio becoming part of the standard library and many third party packages providing features compatible with it, this paradigm is not going away anytime soon.

Let's walk through how to use the HTTPX library to take advantage of this for making asynchronous HTTP requests, which is one of the most common use cases for non-blocking code.

What is non-blocking code?

Asynchronous routines are able to “pause” while waiting on their ultimate result to let other routines run in the meantime.
Asynchronous code, through the mechanism above, facilitates concurrent execution. To put it differently, asynchronous code gives the look and feel of concurrency.

The asyncio library provides a variety of tools for Python developers to do this, and aiohttp provides an even more specific functionality for HTTP requests. HTTP requests are a classic example of something that is well-suited to asynchronicity because they involve waiting for a response from a server, during which time it would be convenient and efficient to have other code running.

Setting up

Make sure to have your Python environment setup before we get started. Follow this guide up through the virtualenv section if you need some help. Getting everything working correctly, especially with respect to virtual environments is important for isolating your dependencies if you have multiple projects running on the same machine. You will need at least Python 3.7 or higher in order to run the code in this post.

Now that your environment is set up, you’re going to need to install the HTTPX library for making requests, both asynchronous and synchronous which we will compare. Install this with the following command after activating your virtual environment:

pip install httpx==0.18.2

With this you should be ready to move on and write some code.

Making an HTTP Request with HTTPX

Let's start off by making a single GET request using HTTPX, to demonstrate how the keywords async and await work. We're going to use the Pokemon API as an example, so let's start by trying to get the data associated with the legendary 151st Pokemon, Mew.

Run the following Python code, and you should see the name "mew" printed to the terminal:

import asyncio
import httpx


async def main():
    pokemon_url = 'https://pokeapi.co/api/v2/pokemon/151'

    async with httpx.AsyncClient() as client:

        resp = await client.get(pokemon_url)

        pokemon = resp.json()
        print(pokemon['name'])

asyncio.run(main())

In this code, we're creating a coroutine called main, which we are running with the asyncio event loop. Here, we are making a request to the Pokemon API and then awaiting a response.

This async keyword basically tells the Python interpreter that the coroutine we're defining should be run asynchronously with an event loop. The await keyword passes control back to the event loop, suspending the execution of the surrounding coroutine and letting the event loop run other things until the result that is being "awaited" is returned.

Making a large number of requests

Making a single asynchronous HTTP request is great because we can let the event loop work on other tasks instead of blocking the entire thread while waiting for a response. This functionality truly shines when trying to make a larger number of requests. Let's demonstrate this by performing the same request as before, but for all 150 of the original Pokemon.

Let's take the previous request code and put it in a loop, updating which Pokemon's data is being requested and using await for each request:

import asyncio
import httpx
import time

start_time = time.time()


async def main():

    async with httpx.AsyncClient() as client:

        for number in range(1, 151):
            pokemon_url = f'https://pokeapi.co/api/v2/pokemon/{number}'

            resp = await client.get(pokemon_url)
            pokemon = resp.json()
            print(pokemon['name'])

asyncio.run(main())
print("--- %s seconds ---" % (time.time() - start_time))

This time, we're also measuring how much time the whole process takes. If you run this code in your Python shell, you should see something like the following printed to your terminal:

8.6 seconds seems pretty good for 150 requests, but we don't really have anything to compare it to. Let's try accomplishing the same thing synchronously.

Comparing speed with synchronous requests

To print the first 150 Pokemon as before, but without async/await, run the following code:

import httpx
import time

start_time = time.time()
client = httpx.Client()

for number in range(1, 151):
    url = f'https://pokeapi.co/api/v2/pokemon/{number}'
    resp = client.get(url)
    pokemon = resp.json()
    print(pokemon['name'])

print("--- %s seconds ---" % (time.time() - start_time))

You should see the same output with a different runtime:

Although it doesn't seem to be that much slower than before. This is most likely because the connection pooling done by the HTTPX Client is doing most of the heavy lifting. However, we can utilize more asyncio functionality to get better performance than this.

Utilizing asyncio for improved performance

There are more tools that asyncio provides which can greatly improve our performance overall. In the original example, we are using await after each individual HTTP request, which isn't quite ideal. We can instead run all of these requests "concurrently" as asyncio tasks and then check the results at the end using asyncio.ensure_future and asyncio.gather.

If the code that actually makes the request is broken out into its own coroutine function, we can create a list of tasks, consisting of futures for each request. We can then unpack this list to a gather call, which runs them all together. When we await this call to asyncio.gather, we will get an iterable for all of the futures that were passed in, maintaining their order in the list. This way we're only using await one time.

To see what happens when we implement this, run the following code:

import asyncio
import httpx
import time


start_time = time.time()


async def get_pokemon(client, url):
        resp = await client.get(url)
        pokemon = resp.json()

        return pokemon['name']


async def main():

    async with httpx.AsyncClient() as client:

        tasks = []
        for number in range(1, 151):
            url = f'https://pokeapi.co/api/v2/pokemon/{number}'
            tasks.append(asyncio.ensure_future(get_pokemon(client, url)))

        original_pokemon = await asyncio.gather(*tasks)
        for pokemon in original_pokemon:
            print(pokemon)

asyncio.run(main())
print("--- %s seconds ---" % (time.time() - start_time))

This brings our time down to a mere 1.54 seconds for 150 HTTP requests! That is a vast improvement over the previous examples. This is completely non-blocking, so the total time to run all 150 requests is going to be roughly equal to the amount of time that the longest request took to run. The exact numbers will vary depending on your internet connection.

Concluding Thoughts

As you can see, using libraries like HTTPX to rethink the way you make HTTP requests can add a huge performance boost to your code and save a lot of time when making a large number of requests.

In this tutorial, we have only scratched the surface of what you can do with asyncio, but I hope that this has made starting your journey into the world of asynchronous Python a little easier. If you're interested in another similar library for making asynchronous HTTP requests, check out this other blog post I wrote about aiohttp.

I’m looking forward to seeing what you build. Feel free to reach out and share your experiences or ask any questions.

Web Scraping and Parsing HTML in Python with Beautiful Soup

Sam Agnew — Fri, 10 Jan 2025 02:52:04 +0000

The internet has an amazingly wide variety of information for human consumption. But this data is often difficult to access programmatically if it doesn't come in the form of a dedicated REST API. With Python tools like Beautiful Soup, you can scrape and parse this data directly from web pages to use for your projects and applications.

Let's use the example of scraping MIDI data from the internet to train a neural network with Magenta that can generate classic Nintendo-sounding music. In order to do this, we'll need a set of MIDI music from old Nintendo games. Using Beautiful Soup we can get this data from the Video Game Music Archive.

Getting started and setting up dependencies

Before moving on, you will need to make sure you have an up to date version of Python 3 and pip installed. Make sure you create and activate a virtual environment before installing any dependencies.

You'll need to install the Requests library for making HTTP requests to get data from the web page, and Beautiful Soup for parsing through the HTML.

With your virtual environment activated, run the following command in your terminal:

pip install requests==2.22.0 beautifulsoup4==4.8.1

We're using Beautiful Soup 4 because Beautiful Soup 3 is no longer being developed or supported.

Using Requests to scrape data for Beautiful Soup to parse

First let's write some code to grab the HTML from the web page, and look at how we can start parsing through it. The following code will send a GET request to the web page we want, and create a BeautifulSoup object with the HTML from that page:

import requests
from bs4 import BeautifulSoup


vgm_url = 'https://www.vgmusic.com/music/console/nintendo/nes/'
html_text = requests.get(vgm_url).text
soup = BeautifulSoup(html_text, 'html.parser')

With this soup object, you can navigate and search through the HTML for data that you want. For example, if you run soup.title after the previous code in a Python shell you'll get the title of the web page. If you run print(soup.get_text()), you will see all of the text on the page. Give it a try!

Getting familiar with Beautiful Soup

The find() and find_all() methods are among the most powerful weapons in your arsenal. soup.find() is great for cases where you know there is only one element you're looking for, such as the <body> tag. On this page, soup.find(id='banner_ad').text will get you the text from the HTML element for the banner advertisement.

soup.find_all() is the most common method you will be using in your web scraping adventures. Using this you can iterate through all of the hyperlinks on the page and print their URLs:

for link in soup.find_all('a'):
    print(link.get('href'))

You can also provide different arguments to find_all, such as regular expressions or tag attributes to filter your search as specifically as you want. You can find lots of cool features in the documentation.

Parsing and navigating HTML with BeautifulSoup

Before writing more code to parse the content that we want, let’s first take a look at the HTML that’s rendered by the browser. Every web page is different, and sometimes getting the right data out of them requires a bit of creativity, pattern recognition, and experimentation.

Our goal is to download a bunch of MIDI files, but there are a lot of duplicate tracks on this webpage as well as remixes of songs. We only want one of each song, and because we ultimately want to use this data to train a neural network to generate accurate Nintendo music, we won't want to train it on user-created remixes.

When you're writing code to parse through a web page, it's usually helpful to use the developer tools available to you in most modern browsers. If you right-click on the element you're interested in, you can inspect the HTML behind that element to figure out how you can programmatically access the data you want.

Let's use the find_all method to go through all of the links on the page, but use regular expressions to filter through them so we are only getting links that contain MIDI files whose text has no parentheses, which will allow us to exclude all of the duplicates and remixes.

Create a file called nes_midi_scraper.py and add the following code to it:

import re

import requests
from bs4 import BeautifulSoup


vgm_url = 'https://www.vgmusic.com/music/console/nintendo/nes/'
html_text = requests.get(vgm_url).text
soup = BeautifulSoup(html_text, 'html.parser')


if __name__ == '__main__':
    attrs = {
        'href': re.compile(r'\.mid$')
    }

    tracks = soup.find_all('a', attrs=attrs, string=re.compile(r'^((?!\().)*$'))

    count = 0
    for track in tracks:
        print(track)
        count += 1
    print(len(tracks))

This will filter through all of the MIDI files that we want on the page, print out the link tag corresponding to them, and then print how many files we filtered.

Run the code in your terminal with the command python nes_midi_scraper.py.

Downloading the MIDI files we want from the webpage

Now that we have working code to iterate through every MIDI file that we want, we have to write code to download all of them.

In nes_midi_scraper.py, add a function to your code called download_track, and call that function for each track in the loop iterating through them:

import re

import requests
from bs4 import BeautifulSoup


vgm_url = 'https://www.vgmusic.com/music/console/nintendo/nes/'
html_text = requests.get(vgm_url).text
soup = BeautifulSoup(html_text, 'html.parser')


def download_track(count, track_element):
    # Get the title of the track from the HTML element
    track_title = track_element.text.strip().replace('/', '-')
    download_url = '{}{}'.format(vgm_url, track_element['href'])
    file_name = '{}_{}.mid'.format(count, track_title)

    # Download the track
    r = requests.get(download_url, allow_redirects=True)
    with open(file_name, 'wb') as f:
        f.write(r.content)

    # Print to the console to keep track of how the scraping is coming along.
    print('Downloaded: {}'.format(track_title, download_url))


if __name__ == '__main__':
    attrs = {
        'href': re.compile(r'\.mid$')
    }

    tracks = soup.find_all('a', attrs=attrs, string=re.compile(r'^((?!\().)*$'))

    count = 0
    for track in tracks:
        download_track(count, track)
        count += 1
    print(len(tracks))

In this download_track function, we're passing the Beautiful Soup object representing the HTML element of the link to the MIDI file, along with a unique number to use in the filename to avoid possible naming collisions.

Run this code from a directory where you want to save all of the MIDI files, and watch your terminal screen display all 2230 MIDIs that you downloaded (at the time of writing this). This is just one specific practical example of what you can do with Beautiful Soup.

The vast expanse of the World Wide Web

Now that you can programmatically grab things from web pages, you have access to a huge source of data for whatever your projects need. One thing to keep in mind is that changes to a web page’s HTML might break your code, so make sure to keep everything up to date if you're building applications on top of this.

If you're looking for something to do with the data you just grabbed from the Video Game Music Archive, you can try using Python libraries like Mido to work with MIDI data to clean it up, or use Magenta to train a neural network with it or have fun building a phone number people can call to hear Nintendo music.

I’m looking forward to seeing what you build. Feel free to reach out and share your experiences or ask any questions.

Working with JSON in Python

Sam Agnew — Mon, 19 Nov 2018 17:36:12 +0000

Often developers need to deal with data in various different formats and JSON, short for JavaScript Object Notation, is one of the most popular formats used in web development. This is the syntax that the JavaScript language uses to denote objects.

As a Python developer, you may notice that this looks eerily similar to a Python dictionary. There are several different solutions to working with JSON in Python, and more often than not this data is loaded into a dictionary.

For this post, we are going to use the following modified JSON data from NASA's Astronomy Picture of the Day API. Navigate to where you want to run the example code, create a file called apod.json and add the following to it:

{
        "copyright": "Yin Hao",
        "date": "2018-10-30",
        "explanation": "Meteors have been shooting out from the constellation of Orion...",
        "hdurl": "https://apod.nasa.gov/apod/image/1810/Orionids_Hao_2324.jpg",
        "media_type": "image",
        "service_version": "v1",
        "title": "Orionids Meteors over Inner Mongolia",
        "url": "https://apod.nasa.gov/apod/image/1810/Orionids_Hao_960.jpg"
}

Using this example, let's examine how you would decode and encode this data with different Python libraries.

The Standard Library

Let's start with the obvious choice, the native JSON module in the Python standard library. This library gets the task of encoding and decoding JSON done in a fairly easy to use way. A lot of the other JSON libraries base their API off of this one and behave similarly.

Create a file called test.py and paste the following code into it to decode the JSON in our apod.json text file, store it in a Python dictionary, and then decode it back into a string:

import json

with open('apod.json', 'r') as f:
        json_text = f.read()

# Decode the JSON string into a Python dictionary.
apod_dict = json.loads(json_text)
print(apod_dict['explanation'])

# Encode the Python dictionary into a JSON string.
new_json_string = json.dumps(apod_dict, indent=4)
print(new_json_string)

Run your code with the following command:

python test.py

One of the upsides about using the built in JSON module is that you don't have to install any third party libraries, allowing you to have minimal dependencies.

simplejson

simplejson is a simple and fast JSON library that functions similarly to the built in module. A cool thing about simplejson is that it is externally maintained and regularly updated.

You will have to install this module with pip. So in your terminal, run the following command (preferably in a virtual environment):

pip install simplejson==3.16.0

This library is designed to be very similar to the built in module, so you don't even have to change your code to get the same functionality! Just import the simplejson module, give it the name json, and the rest of the code from the previous example should just work.

Replace your previous code with the following if you want to use simplejson to encode and decode:

import simplejson as json

with open('apod.json', 'r') as f:
        json_text = f.read()

# Decode the JSON string into a Python dictionary.
apod_dict = json.loads(json_text)
print(apod_dict['explanation'])

# Encode the Python dictionary into a JSON string.
new_json_string = json.dumps(apod_dict, indent=4)
print(new_json_string)

Again, run this with the following command:

python test.py

Many Python developers would suggest using simplejson in place of the stock json library for most cases because it is well maintained.

UltraJSON

Like simplejson, ujson is another community-maintained JSON library. This one, however, is written in C and designed to be really fast. It lacks some of the more advanced features that the built in JSON library has, but really delivers on its promise, as it seems to be unmatched in terms of speed.

Install ujson with the following command:

pip install ujson==1.35

As with simplejson, you don't have to change any of your code for it to work. In most cases, it works in the same way from the developer's point of view as the built in module. Replace your previous code with the following:

import ujson as json

with open('apod.json', 'r') as f:
        json_text = f.read()

# Decode the JSON string into a Python dictionary.
apod_dict = json.loads(json_text)
print(apod_dict['explanation'])

# Encode the Python dictionary into a JSON string.
new_json_string = json.dumps(apod_dict, indent=4)
print(new_json_string)

Run this with the following command:

python test.py

If you're dealing with really large datasets and JSON serialization is becoming an expensive task, then ujson is a great library to use.

The Requests library

These JSON serialization libraries are great, but often in the real world there is more context around why you have to deal with JSON data. One of the most common scenarios that requires decoding JSON would be when making HTTP requests to third party REST APIs.

The requests library is the most popular Python tool for making HTTP requests, and it has a pretty awesome built in json() method on the response object that is returned when your HTTP request is finished. It's great to have a built in solution so you don't have to import more libraries for a simple task.

Install requests with the following shell command:

pip install requests==2.20.0

In this example, we are actually going to make an HTTP request to the Astronomy Picture of the Day API rather than using the local hard coded .json file from the other examples.

Open a new file called apod.py and add the following code to it:

import requests

apod_url = 'https://api.nasa.gov/planetary/apod?api_key=DEMO_KEY'
apod_dict = requests.get(apod_url).json()

print(apod_dict['explanation'])

This code makes an HTTP GET request to NASA's API, parses the JSON data that it returns using this built in method, and prints out the explanation of the current Astronomy Picture of the Day.

Run your code with the following command:

python apod.py

Responding to an HTTP request with JSON in Flask

Another common scenario is that you are building a route on a web application and want to respond to requests with JSON data. Flask, a popular lightweight web framework for Python, has a built in jsonify function to handle serializing your data for you.

Install Flask with pip:

pip install flask==1.0.2

And now create a new file called app.py, where the code for our example web app will live:

from flask import Flask, jsonify

app = Flask(__name__)

@app.route('/apod', methods=['GET'])
def apod():
        url = 'https://apod.nasa.gov/apod/image/1810/Orionids_Hao_960.jpg'
        title = 'Orionids Meteors over Inner Mongolia'

        return jsonify(url=url, title=title)

if __name__ == '__main__':
        app.run()

In this code, we have a route named /apod, and anytime a GET request is sent to that route, the apod() function is called. In this function, we are pretending to respond with the Astronomy Picture of the Day. In this example the data we're returning is just hard coded, but you can replace this with data from any other source.

Run the file with python app.py, and then visit http://localhost:5000/apod in your browser to see the JSON data.

Per the Flask docs, the jsonify function takes data in the form of:

Single argument: Passed straight through to dumps().
Multiple arguments: Converted to an array before being passed to dumps().
Multiple keyword arguments: Converted to a dict before being passed to dumps().
Both args and kwargs: Behavior undefined and will throw an exception.

This function wraps dumps() to add a few enhancements that make life easier. It turns the JSON output into a Response object with the application/json mimetype.

Conclusion

There are many different solutions to working with JSON in Python, and I've shown you just a few examples in this post. You can use whichever library suits your personal needs, or in the case of requests and Flask, might not even have to import a specific JSON library.

Feel free to reach out for any questions or to show off what you build:

Email: Sagnew@twilio.com
Twitter: @Sagnewshreds
GitHub: Sagnew
Twitch (streaming live code): Sagnewshreds