DEV Community: Eric Dong

Ship Collisions

Eric Dong — Tue, 30 Jan 2024 02:40:37 +0000

This marks the final installment of my blog series covering the v0.4 release of Space War RL! With these simulation updates concluded, my focus will shift towards Reinforcement Learning in future posts.

Collision Resolution Process

The general process is to first detect when the ship collides with another ship and to adjust their velocities accordingly.

Initially, I attempted to resolve collisions by swapping the velocities of the colliding ships:

self_vel_x, self_vel_y = self.vel
other_vel_x, other_vel_y = sprite.vel

# Swap velocities
new_self_vel_x = other_vel_x
new_self_vel_y = other_vel_y
new_other_vel_x = self_vel_x
new_other_vel_y = self_vel_y

self.vel = (new_self_vel_x, new_self_vel_y)
sprite.vel = (new_other_vel_x, new_other_vel_y)

Ship collisions v0: Represents a perfect system with no energy loss.

However, I later opted to introduce energy loss by reducing the other ship's velocity by 50%:

new_self_vel_x = other_vel_x * 0.50
new_self_vel_y = other_vel_y * 0.50
new_other_vel_x = self_vel_x * 0.50
new_other_vel_y = self_vel_y * 0.50

Ship collisions v1: Gain 50% of the other ship's velocity

My physics friend corrected me, explaining that momentum should be conserved, implying that the ship should continue moving forward after colliding with another ship. This looks more natural!

new_self_vel_x = (self_vel_x * 0.2) + (other_vel_x * 0.75)
new_self_vel_y = (self_vel_y * 0.2) + (other_vel_y * 0.75)
new_other_vel_x = (other_vel_x * 0.2) + (self_vel_x * 0.75)
new_other_vel_y = (other_vel_y * 0.2) + (self_vel_y * 0.75)

Ship collisions v2: Maintain 20% of original velocity and gain 75% of the other ship's velocity

In the future, I may want to take mass into account for collisions.

Handling Sprite Overlap

A major and unexpected problem I encountered was when the ships overlapped due to a "low-force" collision.

OH NO!

I created this util function to determine the amount of overlap. I compare the center point of each colliding sprites to compute the direction of collision.

def check_overlapping_sprites(
    sprite: pygame.sprite.Sprite, sprite_other: pygame.sprite.Sprite
):
    """Returns the horizontal and vertical overlap between two sprites."""
    overlap_x, overlap_y = (0, 0)
    if sprite.rect.centerx < sprite_other.rect.centerx:
        overlap_x = sprite.rect.right - sprite_other.rect.left
    elif sprite.rect.centerx > sprite_other.rect.centerx:
        overlap_x = -(sprite_other.rect.right - sprite.rect.left)

    if sprite.rect.centery < sprite_other.rect.centery:
        overlap_y = sprite.rect.bottom - sprite_other.rect.top
    elif sprite.rect.centery > sprite_other.rect.centery:
        overlap_y = -(sprite_other.rect.bottom - sprite.rect.top)

    return overlap_x, overlap_y

Perhaps there is a better way, but this will work for now.

If I update the position of the sprites to avoid overlap, the sudden movement looks jarring and unnatural.

# detect any overlap and move the ships
overlap_x, overlap_y = check_overlapping_sprites(self, sprite)

sprite.pos = (sprite.pos[0] + overlap_x, sprite.pos[1] + overlap_y)

Sprite overlap fix v1

Looking at the emulator for comparison, I noticed it is doing something similar.

Ship collisions in Emulator

Both of the ships' positions are adjusted. I could do something similar where I apply half of the overlap to the first sprite and the other half to the second sprite, but I didn't want to do this.

For a more natural looking approach, if there is any overlap, move the sprites naturally by using their velocities.

# detect any overlap and move the ships
overlap_x, overlap_y = check_overlapping_sprites(self, sprite)

if overlap_x:
    sprite.pos = (sprite.pos[0] + sprite.vel[0], sprite.pos[1])
if overlap_y:
    sprite.pos = (
        sprite.pos[0],
        sprite.pos[1] + sprite.vel[1],
    )

Sprite overlap fix v2. Note that angular velocities and collisions are not considered in this simulation.

This looks much better in my opinion. Instead of updating the ships' positions immediately, I can move them away from each other using their new velocities after collision. This continues until there is no overlap.

My goal is not to make a perfect physics simulation, so this is sufficient. I may use Unreal or Unity in the future to take advantage of the physics engine.

Next Up

I will be doing a mini-post on using github actions to automate deployment of my simulation to itch.io. After that, I will be drawing pixel art for the player 2 ship and preparing to train my agent using the DQN algorithm. I will see how much I can learn and implement from attempting to derive everything from Playing Atari with Deep Reinforcement Learning.

Adding Weapons

Eric Dong — Thu, 25 Jan 2024 01:55:16 +0000

This is my blog part 2 of 3 for the v0.4 release of Space War RL!

Making a Common Base Class

The HumanShip class contains logic for applying screen wrap-around, velocity, and rotation. Since I want the other objects to have this same logic, I can extract this into a common base class.

class SpaceEntity(pygame.sprite.Sprite):
   ...

    def update(self, *_args, **kwargs):
        """Entrypoint for updating the player state each frame

        screen_wrap, pos, and rotation updates can be disabled if set to False
        """
        x_pos, y_pos = self.pos
        x_vel, y_vel = self.vel

        # conditions for wrapping around the screen
        if x_pos >= SCREEN_WIDTH:
            x_pos = 0
        elif x_pos <= 0:
            x_pos = SCREEN_WIDTH
        if y_pos >= SCREEN_HEIGHT:
            y_pos = 0
        elif y_pos <= 0:
            y_pos = SCREEN_HEIGHT

        # apply velocity to position
        x_pos += x_vel
        y_pos += y_vel
        self.pos = (x_pos, y_pos)
        self.rect.center = self.pos

        # update rotation to surface
        self.ang %= 360
        self.image = pygame.transform.rotate(self.surf, -self.ang)
        self.rect = self.image.get_rect(center=self.pos)

The SpaceEntity base class represents flying objects in space.
See space_war/sim/base.py for the full source

I also have a base class to represent a weapon.

class BaseWeapon(pygame.sprite.Sprite):
    """Simple base class for weapons. All weapons have a specific duration."""

    def __init__(self, duration) -> None:
        pygame.sprite.Sprite.__init__(self)
        self.start_time = pygame.time.get_ticks()
        self.max_duration = duration

    def is_expired(self) -> bool:
        """Returns True if duration of weapon exceeds the max duration"""
        current_duration = pygame.time.get_ticks() - self.start_time
        return current_duration >= self.max_duration

    def update(self, *args: Any, **kwargs: Any):
        """Update entrypoint for weapons"""
        super().update(*args, **kwargs)
        if self.is_expired():
            self.kill()

See space_war/sim/weapon.py for the full source

Adding Photon Torpedoes

The photon torpedoes were pretty easy to implement, since I could inherit the logic from my base class.

The graphics in the original were boring, so I drew my own torpedo.

Screenshot of photon torpedoes from emulator

Pygame comes with some utils to draw lines and polygons:

pygame.draw.polygon(self.image, "white", [[5, 0], [3, 5], [7, 5]], 1)
pygame.draw.polygon(self.image, "white", [[0, 10], [3, 9], [3, 5]], 1)
pygame.draw.polygon(self.image, "white", [[7, 5], [7, 9], [11, 10]], 1)
pygame.draw.line(self.image, "white", (3, 9), (7, 9))
pygame.draw.line(self.image, "white", (5, 5), (5, 11))

Closeup of firing Photon Torpedoes

Each torpedo flies for its max duration unless it hits another object. There can only be a max of 7 per ship flying at a time (According to the emulator).

Adding Phasers

Adding phasers on the other hand was more complicated than I initially thought. I ended up making 3 revisions of the implementation. I started using the SpaceEntity base class and realized screen wrap-around won't work correctly. The base class assumes there is one rectangle. This works fine to represent individual ships and torpedoes. For screen wrap-around to work correctly, I need at least 2 rectangles. One is drawn to the edge of the screen and the rest continues on the other side of the screen. Another big difference from the ship and torpedo is the size of the surface. The phaser's surface is the same size as the screen. This made the calculations easier for drawing the line, since I could use absolute coordinates. The torpedo is a 12x12 surface and it is rotated based on the ship's angle upon firing.

My implementation consists of three steps:

Create invisible hit detector rectangles
Detect collisions with ships or other torpedoes
Draw the phaser line

Closeup of firing Phasers

Putting Everything Together

Firing weapons are handled by adding a new instance of the weapon to its sprite group to manage rendering and updates.
The ship's handle_events function was getting long, so I refactored this method to call two helper methods, _handle_movement_events and _handle_firing_weapon_events.

This is what the _handle_firing_weapon_events function looks like:

def _handle_firing_weapon_events(self, event: pygame.event.Event):
    """Handles firing phasers and photon torpedoes.

    Keybindings are hard coded for now. It would be good to make this config driven.

    """
    if event.type == pygame.KEYDOWN:
        if event.key == pygame.constants.K_q:
            pygame.time.set_timer(
                self.fire_phaser_repeat_event,
                PHASER_FIRE_CD,
            )
            if not self._check_phaser_on_cooldown():
                self.phaser = Phaser(source_ship=self)
                self.phaser_group.add(self.phaser)
                self.phaser_last_fired = pygame.time.get_ticks()
        if event.key == pygame.constants.K_e:
            pygame.time.set_timer(
                self.fire_torpedoes_repeat_event,
                TORPEDO_FIRE_CD,
            )
            if (
                len(self.torpedo_group) < MAX_TORPEDOES_PER_SHIP
                and not self._check_torpedo_on_cooldown()
            ):
                self.torpedo_group.add(
                    PhotonTorpedo(
                        start_pos=self.pos,
                        start_ang=self.ang,
                        start_vel=self.vel,
                    )
                )
                self.torpedo_last_fired = pygame.time.get_ticks()

    if event.type == pygame.KEYUP:
        if event.key == pygame.constants.K_q:
            pygame.time.set_timer(self.fire_phaser_repeat_event, 0)
        if event.key == pygame.constants.K_e:
            pygame.time.set_timer(self.fire_torpedoes_repeat_event, 0)

    if event.type == self.fire_phaser_repeat_event:
        self.phaser = Phaser(source_ship=self)
        self.phaser_last_fired = pygame.time.get_ticks()
        self.phaser_group.add(self.phaser)
    if event.type == self.fire_torpedoes_repeat_event:
        if len(self.torpedo_group) < MAX_TORPEDOES_PER_SHIP:
            self.torpedo_last_fired = pygame.time.get_ticks()
            self.torpedo_group.add(
                PhotonTorpedo(
                    start_pos=self.pos,
                    start_ang=self.ang,
                    start_vel=self.vel,
                )
            )

See space_war/sim/ship.py for the full source

The final step is to detect collisions between the weapons and ships. For this iteration, one hit with a weapon will result in death (removing the sprite from its group). Torpedoes can collide with other torpedoes and ships. Phasers can only collide with torpedoes and the other ship.

GIF of weapons demo

Lessons Learned

While implementing the weapons, I have a better understanding how inheritance works in python and the difference between the surface and rectangle. The surface is the visible portion of drawing pixels to the screen and the rectangle is the coordinate data. In this case, I'm using the rectangle for movement and collision detection. I also learned more about sprite groups and how it makes it easy to manage multiple sprite instances. You can see space_war/main.py for how I'm initializing the groups.

Next Steps

My next blog will show some physics I added for ship on ship collisions! Thank you for reading.

Playable Sandbox Now Available

Eric Dong — Fri, 12 Jan 2024 01:48:18 +0000

This is my blog part 1 of 3 for the v0.4 release of Space War RL!

Normally games made with pygame are not playable from the web. They can only be run from the command line or use PyInstaller or cx_Freeze to create a standalone executable.

I recently discovered pygbag that allows python code to run in a web browser.

You can play my stable version here. You can play my dev version here.

Stable Version

This build will follow my latest github releases/tags.

Dev Version

This build will serve as my testing grounds for new features to solicit more feedback. At the time of writing this post, the dev version contains weapons, ship collisions, and more. My next 2 blog posts in this series will cover my updates.

What is pygbag?

Pygbag is a C runtime linked to cython-wasm compiled to WebAssembly by Emscripten and hosted on pygame-web.github.io. You can read more about pygbag on their wiki page. You can see some demos here.

Issues/Challenges I faced

Patching pygame.time.set_timer

Due to pygbag#16, the built-in function does not work correctly in the pygame-wasm environment. I noticed the patch to set_timer was buggy, so I made an improvement to the patch by handling other behaviors mentioned in the docs (e.g. disabling the timer and discarding an old timer for the same event).

Implementation Details

I created a THREADS dictionary to keep track of all the spawned threads. It is keyed by the event type and contains the uuid of the thread. A delay of 0 will cancel the timer by deleting the event type from the dictionary, which causes the while loop to break in the fire_event function. If there are multiple threads for the same event type, only the latest one is considered due to this condition in the if block: THREADS[event] != thread_uuid.

This is only a temporary workaround until a fix is released upstream. See below for the code example:

def patch_timer():
    import asyncio
    import aio.gthread

    # Global var to keep track of timer threads
    #   - key: event type
    #   - value: thread uuid
    THREADS = {}

    def patch_set_timer(
        event: Union[int, pygame.event.Event], millis: int, loops: int = 0
    ):
        """Patches the pygame.time.set_timer function to use gthreads"""

        dlay = float(millis) / 1000
        cevent = pygame.event.Event(event)
        event_loop = asyncio.get_event_loop()

        async def fire_event(thread_uuid):
            """The thread's target function to handle the timer

            Early exit conditions:
            - event loop is closed
            - event type is no longer in THREADS dictionary
            - the thread's uuid is not the latest one
            - Max loop iterations if loops param is not zero
            """
            loop_counter = 0
            while True:
                await asyncio.sleep(dlay)
                if (
                    event_loop.is_closed()
                    or event not in THREADS
                    or THREADS[event] != thread_uuid
                    or (loops and loop_counter >= loops)
                ):
                    break

                pygame.event.post(cevent)
                loop_counter += 1 if loops else 0

        if dlay > 0:
            # uuid is used to track the latest thread,
            # stale threads will be terminated
            thread_uuid = uuid.uuid4()
            Thread(target=fire_event, args=[thread_uuid]).start()
            THREADS[event] = thread_uuid

        else:
            # This cancels the timer for the event
            if event in THREADS:
                del THREADS[event]

    pygame.time.set_timer = patch_set_timer


if platform.system().lower() == "emscripten":
    patch_timer()

Adding Physics and Finalizing Player Movement

Eric Dong — Sat, 11 Nov 2023 03:40:04 +0000

This is an update for v0.3 of Space War RL! You can see a summary of my changes in my releases. You can view my kanban here.

Making Ship Rotation more Discrete

In the emulator, I noticed each keypress of the rotation keybinding would rotate the ship by 22.5 degrees (4 key presses for a 90 degree turn).
The current rotation handling logic is being called from the player's update function, which gets called every frame. I was using pygame.key.get_pressed, which is only used to check which keys are currently pressed down. What I want is to detect when the rotation key is pressed down and increment or decrement the angle by 22.5. I will need to check the rotation keybinding in the event loop.

main.py

while running:
    # event loop
    # pygame.QUIT event means the user clicked X to close your window
    for event in pygame.event.get():
        player_1.handle_events(event)
        if event.type == pygame.QUIT:
            running = False

player.py

def handle_events(self, event: Event):
    """Handles keyboard input to update ship's rotation and position state"""
    if event.type == KEYDOWN:
        ...
        if event.key == K_a:
            self.ang -= 22.5
        if event.key == K_d:
            self.ang += 22.5
        self.ang = self.ang % 360

I also added pygame.key.set_repeat(100, 100) in main.py#L14 to allow the KEYDOWN event to be repeated every 100ms, so I can hold down the rotation keybinding for multiple rotations. Otherwise it is only triggered once per keypress.

Demo

Made ship rotation more discrete, rotations are made in 22.5 degree increments

A possible alternate implementation is using numpy and indexing through the array.

>>> import numpy as np
>>> np.arange(0, 360, 22.5)
array([  0. ,  22.5,  45. ,  67.5,  90. , 112.5, 135. , 157.5, 180. ,
       202.5, 225. , 247.5, 270. , 292.5, 315. , 337.5])

I noticed my implementation didn't handle multiple keypresses and release correctly. I took a look at the emulator and it's not perfect either.

Holding down A first and then holding down D correctly switches the player's rotation from counterclockwise to clockwise. Holding down D first and then pressing A DOES NOT switch the rotation from clockwise to counterclockwise

It looks like the check for D is overriding the check for A. This is the pseudocode I'm thinking that's causing this behavior in the emulator.

IF (D is down) THEN
    rotate clockwise
ELSE IF (A is down) THEN
    rotate counterclockwise

The check for D appears to be first causing inconsistent behavior. I want to see if I can check my Dad's computer to see if this is in the original game.

I decided not to copy this behavior and fixed it in my implementation.

def handle_events(self, event: Event, check_key_event: Event):
    """Handles keyboard input to update ship's rotation and position
    state
    """
    if event.type == KEYDOWN:
        # Handle ship movement and rotation
        if event.key == K_a:
            self.rotate_ccw_lock = True
        if event.key == K_d:
            self.rotate_ccw_lock = False
    if event.type == KEYUP:
        if event.key == K_a:
            self.rotate_ccw_lock = False
        if event.key == K_d:
            self.rotate_ccw_lock = True
    if event.type == check_key_event:
        keys = key.get_pressed()
        rotate_ccw = keys[K_a] and self.rotate_ccw_lock
        rotate_cc = keys[K_d] and not self.rotate_ccw_lock
        ...
        # Update rotation
        if rotate_ccw:
            self.ang -= 22.5
        elif rotate_cc:
            self.ang += 22.5
        self.ang %= 360

I am using the the self.rotate_ccw_lock field as a way to detect when the A or D is first pressed or released.

For input check polling, I replaced the pygame.key.set_repeat function with a custom user event to check user input.

# create custom event to check user input
check_key_event = pygame.USEREVENT + 1
pygame.time.set_timer(check_key_event, CHECK_KEYS_TIME_DELAY_MS)
...

while running:
    # event loop
    for event in pygame.event.get():
        player_one.handle_events(event, check_key_event)

Demo

Rotation Fix

Adding Zero Gravity Physics

From my previous blog post, I discovered this formula to calculate the new relative x and y coordinates:

delta_y = dist * sin(angle)
delta_x = dist * cos(angle)

To add velocity, the player just needs to keep track of its net x and y velocity and each frame would add the net velocities to the player's current position.

def handle_events(self, event: Event, check_key_event: Event):
    """Handles keyboard input to update ship's rotation and position
    state
    """
    ...
    if event.type == check_key_event:
        keys = key.get_pressed()
        ...
        accelerate = keys[K_s]

        # Update acceleration
        if accelerate:
            self.x_vel += math.cos(self.ang * math.pi / 180)
            self.y_vel += math.sin(self.ang * math.pi / 180)
    ...

def update(self, *args, **kwargs):
    """Entrypoint for updating the player state each frame"""
    x_pos, y_pos = self.pos

    ...

    # Apply velocity to position
    x_pos += self.x_vel
    y_pos += self.y_vel
    self.rect.x = x_pos
    self.rect.y = y_pos
    self.pos = (x_pos, y_pos)

    ...

After making these changes, I noticed a design pattern has emerged.

In the event loop, the handle_events function is used to update the player's internal state from user input. E.g. calculate changes in the net x and y velocity as well as the angle.
The update function is used to apply the values of the internal state variables to the environment. E.g. updating the player's position from net velocity and applying rotation to the surface from the angle.

Demo

Applying velocity instead of a fixed distance for movement. At the end of the gif, the player moves to the upper right and then out of bounds.

Whoops, it is easy to go out of bounds now! I will be adding screen wrap-around for this for now, since that is present in the original game. In other environment simulations, the bounds are restricted, so it is like the agent is in a box. Trying to move beyond the boundaries will place the agent at the boundary. Another method in RL is to punish the agent from going out of bounds via a negative reward. The agent can still go out of bounds, but hopefully it learns not to do that with training. The latter is more realistic.

Adding Screen Wrap-Around

def update(self, *args, **kwargs):
    """Entrypoint for updating the player state each frame"""
    x_pos, y_pos = self.pos

    # Conditions for wrapping around the screen
    if x_pos >= SCREEN_WIDTH + 10:
        x_pos = 0
    elif x_pos <= -10:
        x_pos = SCREEN_WIDTH
    if y_pos >= SCREEN_HEIGHT + 10:
        y_pos = 0
    elif y_pos <= -10:
        y_pos = SCREEN_HEIGHT

Demo

Added screen wrap-around to prevent the ship from going out of bounds

Thank you for reading my blog! My next blog post will discuss adding weapons to the agent.

Getting Started and Adding Player Movement

Eric Dong — Wed, 08 Nov 2023 04:37:18 +0000

Selecting the Appropriate Simulator

I was deciding whether I should use game engines like Godot, Unity, or Unreal, but ended using pygame. ML libraries are primarily in python, so pygame and OpenAI gym are a good choice. Space War also has simple graphics, so I don't need the overhead of the game engine. I ended up going with pygame because I only used gym at work and wanted to try something new. pygame also seemed easier to add my own graphics, add human controls, and manage collisions (e.g. implement damage dealt to player from weapons) compared to gym. However if I want to make the simulation more complex and add game-like features, I will revisit game engines. I think having a level editor would be helpful if I want to test if RL agents can generalize to different environments. I may also try using a game engine to train RL agents in 3D environments, but that's for a future project!

Selecting the Appropriate ML Framework

I will be using pytorch because I used that ML framework in the past. Others have said pytorch is more pythonic than tensorflow as well.

Learning Pygame

I first discovered pygame when I found this pygame project using tensorflow at work, but ran out of time to dig deeper into his implementation. These are the steps I took to start learning more about pygame.

The pygame documentation recommend this video tutorial and I totally agree. The content is well organized and presented clearly. I didn’t code along with the author, but I looked at what parts were relevant and applied it to my own game.
Looked up documentation on the Sprite module.
Used the quick-start code from the pygame documentation

# Example file showing a circle moving on screen
import pygame

# pygame setup
pygame.init()
screen = pygame.display.set_mode((800, 600))
clock = pygame.time.Clock()
running = True
dt = 0

player_pos = pygame.Vector2(screen.get_width() / 2, screen.get_height() / 2)

while running:
    # poll for events
    # pygame.QUIT event means the user clicked X to close your window
    for event in pygame.event.get():
        if event.type == pygame.QUIT:
            running = False

    # fill the screen with a color to wipe away anything from last frame
    screen.fill("black")

    pygame.draw.circle(screen, "white", player_pos, 40)

    keys = pygame.key.get_pressed()
    if keys[pygame.K_w]:
        player_pos.y -= 300 * dt
    if keys[pygame.K_s]:
        player_pos.y += 300 * dt
    if keys[pygame.K_a]:
        player_pos.x -= 300 * dt
    if keys[pygame.K_d]:
        player_pos.x += 300 * dt

    # flip() the display to put your work on screen
    pygame.display.flip()

    # limits FPS to 60
    # dt is delta time in seconds since last frame, used for framerate-
    # independent physics.
    dt = clock.tick(60) / 1000

pygame.quit()

Demo

Using quickstart code to move a white circle using the W-A-S-D keys.

Adding Player One Pixel Art

Since the graphics in space war is simple (all white, no shaders), I drew my own pixel art by copying what player 1 looked like.

I'm no artist, so I just searched reddit to see pixel art apps that supports Linux and stumbled upon mtpaint. The player ships were symmetric, so I just drew the top half and mirrored it to make the bottom half. Let me know if you have a favorite pixel art tool!

Screenshot of player one ship in mtpaint

Replacing Circle with Ship Image

I added a separate Player class that extends from the Sprite class to manage its state.

class Player(Sprite):
    """Represents the human player"""

    def __init__(self, delta_time, image_path, start_pos) -> None:
        super().__init__()
        self.surf = image.load(image_path).convert_alpha()
        self.image = self.surf
        self.pos = start_pos
        self.rect = self.surf.get_rect(midbottom=self.pos)
        self.delta_time = delta_time

    def handle_input(self):
        """Handles keyboard input to update the player's position"""
        keys = key.get_pressed()
        x_pos, y_pos = self.pos
        if keys[K_w]:
            y_pos -= 0.05 * SCREEN_HEIGHT * self.delta_time
        if keys[K_s]:
            y_pos += 0.05 * SCREEN_HEIGHT * self.delta_time
        if keys[K_a]:
            x_pos -= 0.05 * SCREEN_WIDTH * self.delta_time
        if keys[K_d]:
            x_pos += 0.05 * SCREEN_WIDTH * self.delta_time
        self.pos = (x_pos, y_pos)
        self.rect.x = x_pos
        self.rect.y = y_pos

    def update(self):
        """Entrypoint for updating the player state each frame"""
        self.handle_input()

main game loop:

def main():
    """Entrypoint for starting up the pygame"""
    # pygame setup
    pygame.init()
    screen = pygame.display.set_mode((SCREEN_WIDTH, SCREEN_HEIGHT))

    # delta time in seconds since last frame, used for framerate-
    # independent physics.
    clock = pygame.time.Clock()
    delta_time = clock.tick(MAX_FPS) / 1000

    # Groups
    player_single_group = pygame.sprite.GroupSingle()
    player_single_group.add(
        Player(
            delta_time=delta_time,
            image_path=module_path() / "assets" / "player_1.png",
            start_pos=(screen.get_width() / 2, screen.get_height() / 2),
        )
    )

    running = True

    while running:
        # event loop
        # pygame.QUIT event means the user clicked X to close your window
        for event in pygame.event.get():
            if event.type == pygame.QUIT:
                running = False

        # fill the screen with a color to wipe away anything from last frame
        screen.fill("black")

        player_single_group.draw(screen)
        player_single_group.update()

        pygame.display.update()

    pygame.quit()

Demo of v0.1

Using a sprite to replace the white circle with the ship image

Updating Player Movement

The ship should be able to rotate and only move in the direction it is facing.

I added a field to keep track of the angle of the ship and updated the if statements like so:

if keys[K_a]:
    self.ang -= 50 * self.delta_time
if keys[K_d]:
    self.ang += 50 * self.delta_time
...
self.ang = self.ang % 360

Next, the surface needs to be rotated so we can update the sprite's image.

# Update rotation
current_rect = self.surf.get_rect()
newsurf = transform.rotate(self.surf, -self.ang)
newrect = newsurf.get_rect()
# put new surface rect center on same spot as old surface rect center
self.rect.x += current_rect.centerx - newrect.centerx
self.rect.y += current_rect.centery - newrect.centery
self.image = newsurf

For forward movement, I removed the handler for the w key and updated the s key.

x_pos, y_pos = self.pos
if keys[K_s]:
    x_pos += 30 * self.delta_time * math.cos(self.ang * math.pi / 180)
    y_pos += 30 * self.delta_time * math.sin(self.ang * math.pi / 180)
...
self.pos = (x_pos, y_pos)
self.rect.x = x_pos
self.rect.y = y_pos

The formula is derived from SOHCAHTOA. The sine of theta (the angle) is the length of opposite side / length of hypotenuse and the cosine of theta is length of adjacent side / length of hypotenuse.

In the context of this simulation:

theta is the angle the player is facing (0 - 359)
opposite side is the change in y
adjacent side is the change in x
hypotenuse is the distance traveled.

Therefore I can calculate the new player's relative position via

delta_y = dist * sin(angle)
delta_x = dist * cos(angle)

The angle is multiplied by pi/180 to convert to radians.

I'm glad I remembered my high school trigonometry!

Demo of v0.2

Ship can rotate with A-D keys (counterclockwise and clockwise respectively) and move in the current direction it is facing with the 'S' key. When 'S' is released, the ship stops moving forward.

Thank you for reading my blog! The next blog post will be on adding zero gravity physics to the environment.

Series Introduction

Eric Dong — Sat, 04 Nov 2023 03:12:07 +0000

Machine learning is a field I always wanted to get into. I always thought I would need a masters or a PHD to get started; Something I thought was out of my reach...until recently.

This year I had the opportunity to get my hands dirty; I got involved with a group that specializes in machine learning. I worked on some Reinforcement Learning (RL) projects at work, but only scratched the surface due to limited funding. Regardless, this was the spark that I was missing! Since there is limited opportunity at work, I must carve a path for myself on my own.

This project will allow me to express my creativity, dive deeper, discover new things, and most importantly have fun!

Inspiration

This project combines nostalgia with some of my interests: gaming, coding, and machine learning!

I remembered my dad showed me the 1985 Bill Seiler's version of space War on his IBM PC/AT as a child.

You can play the emulator from your browser on internet archive

I remembered the controls are hard for the human player, so I'm curious if Reinforcement Learning agents can play the game better.

First I will start building out the simulation/game until I have a minimal viable product (MVP) to start training the agents.

I will also recreate the original "AI" to measure baseline performance. I assume they are rules-based and have access to internal state. The left player was defensive and mostly used phasers. The right player was offensive and mostly used photon torpedoes.

My metrics of performance will be the game win rate. Once I get this going I will continue to make the simulation more complex and add more agents for Multi Agent Reinforcement Learning (MARL).

Roadmap

Crawl. Walk. Run. Let's start simple and slowly work things up! My initial plan is to iterate on the simulation, iterate the agents, and rinse and repeat.

You can see what I'm working on with my github project

Simulation v0

Some features will be omitted for simplicity in order to start working with RL sooner.

These features will be implemented in this first iteration:

Basic movement with zero gravity physics
Screen wrap-around
Weapons (Photon torpedoes and Phasers)
Energy management (balance shield and energy)

Baseline Agents

I will implement the defensive and offensive rule-based agents, denoted by baseline_d and baseline_o respectively. This will allow me to fix any bugs with the simulation prior to training the RL agents.

RL Agents

I will start with the classic Deep Q Network (DQN)¹ using a continuous state space and discrete action space. I will be trying to implement this from scratch with the paper rather than copying code from a tutorial.

I will train the DQN agent against 3 agents to start:

baseline defensive, denoted DQN_d
baseline offensive, denoted DQN_o
itself using competitive self-play, denoted DQN_s. The article mentioned the agents were initally trained with dense rewards to aid in exploration and then gradually replaced with sparse rewards when winning/losing the game. I will try using constant sparse rewards to see what happens first.

This will allow me to run my first experiment. I will compare the win rate between these pairs:

baseline_o vs. baseline_d
DQN_d vs. baseline_d
DQN_o vs. baseline_d
DQN_s vs. baseline_d
DQN_d vs. baseline_o
DQN_o vs. baseline_o
DQN_s vs. baseline_o
DQN_o vs. DQN_d
DQN_s vs. DQN_d
DQN_s vs. DQN_o

My initial hypothesis is the agents will overfit to the agent they were training against.
The agent they trained against will perform well during evaluation, but probably perform poorly when faced with different opponents. If I am right, I will need to train the DQN agents with a variety opponents in order for them to discover a generic strategy.

Simulation v1

Implement the other features in Space war 1985.

Gravity
Planet
Give Agent ability to warp via hyper space
Give Agent ability to cloak

RL Agents

Rerun experiments and see how it impacts training and evaluation. I will also look into other algorithms like PPO and DDPG.

Simulation v2

Once I have a good understanding of how the agents behave in a 1v1 setting, I will see what happens in team battles. I found this OpenAI article and paper² fascinating because agents can learn to cooperative with each other in order to "outsmart" the other team. The other team can then learn a counter strategy. Both teams alternate in giving the other harder tasks. This feedback loop results in an evolutionary arms race where each team creates an implicit learning curriculum for each other. This training method allows agents to reach superhuman performance like with Dota 2³ and Starcraft II⁴. To learn more about OpenAI's Dota 2 research see this article.

Simulation v3

The final step is to see how to make the simulation more realistic and push the boundaries of RL to the limit!

Based on this roadmap, my initial posts in this series will be a software engineering effort to build the simulation and then RL research and tuning. More updates to come soon!

Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., & Riedmiller, M. "Playing Atari with Deep Reinforcement Learning" arXiv, 2013. Available: https://doi.org/10.48550/arXiv.1312.5602 ↩
Baker, B., Kanitscheider, I., Markov, T., Wu, Y., Powell, G., McGrew, B., & Mordatch, I. "Emergent Tool Use From Multi-Agent Autocurricula" arXiv, 2020. Available: https://doi.org/10.48550/arXiv.1909.07528 ↩
OpenAI, "Dota 2 with large scale deep reinforcement learning" arXiv, 2019, Available: https://doi.org/10.48550/arXiv.1912.06680 ↩
Vinyals, O., Babuschkin, I., Chung, J., Mathieu, M., Jaderberg, M., et al. AlphaStar: Mastering the real-time strategy game StarCraft II. https://deepmind.com/blog/alphastar-mastering-real-time-strategy-game-starcraft-ii/, 2019. ↩

Introduction

Eric Dong — Sat, 28 Oct 2023 22:43:19 +0000

Hi everyone! I'm a software engineer currently living in Pittsburgh with my wife, dog, and two kids. When I'm not with my family and friends, I like to learn chinese, configure linux pcs, and write scripts/code.

I grew up in upstate New York and studied at the University of Pittsburgh for my undergraduate from 2014 - 2018. I originally majored in neuroscience, but switched to Computer Science in the spring of my sophomore year. I still wanted to graduate with my friends, so I worked really hard by taking summer classes (3 - 4 CS classes every semester). The amount of projects, assignments, and exams were crazy! If I could redo my undergrad, I would start taking CS courses in freshman year rather than compressing everything into two and a half years.

I posted a couple years ago on Medium on web development, but I started to lose interest. I was primarily working with a lot of JavaScript and react-redux projects at work at the time and it started getting repetitive. A lot has changed in the past 5 years; I learned backend development, distributed programming, docker, gitlab CI, python, and some machine learning. I also started to realize there is less freedom at work, so I decided to work on more personal projects as hobbies.

First PC Build (2018)

My parents were strict on gaming, so I only played free games on parent's underpowered desktop with slow internet. So the first thing I did after I got my first job after college, was build a PC. This is an expensive hobby, so I will probably swap out parts rather than building a whole new PC.

These were my hardware specs:

Hardware	Part
CPU	Intel - Core i7-8700K 3.7GHz 6-Core Processor
CPU Cooler	Cooler Master - Hyper 212 EVO 82.9 CFM Sleeve Bearing CPU CooMOBOler
MOBO	Asus - ROG MAXIMUS X HERO (WI-FI AC) ATX LGA1151 Motherboard
RAM	Corsair - Vengeance LPX 16GB (2 x 8GB) DDR4-2400 Memory
SSD	Samsung - 850 EVO-Series 500GB 2.5" Solid State Drive
HDD	Seagate - Barracuda 2TB 3.5" 7200RPM Internal Hard Drive
GPU	MSI - GeForce GTX 1080 8GB Video Card
CHASSIS	Fractal Design - Define R6 White TG ATX Mid Tower Case
PSU	Corsair - 1000W 80+ Platinum Certified Fully-Modular ATX Power Supply

In 2023, I switched out my 1080 gtx with a 3070 rtx.

Linux Journey (2018 - Present)

I decided to take Linux more seriously after the announcement of Windows 11. The hardware requirement of the TPM seemed non standard to me for non-enterprise pcs and I was already tired of Microsoft shenanigans and lack of user configuration. I noticed this weird alternating pattern of "good" versions of windows (XP - good, Vista - bad, Windows 7 - good, Windows 8 - bad, Windows 10 - Good, Windows 11 - bad?). I was at my breaking point, it was time to switch...

My first Linux distro was PopOS and It was a refreshing experience. It was really easy to install, use, and game on. I distro hopped a few times to see what other linux flavors are like. These are the ones that I remember trying Zorin OS and Linux Mint. These ones looked mostly like windows and it was easy to use. At work, I gained most of my linux knowledge from docker and configuring / administrating RHEL servers. This helped give me more confidence in switching my personal computers to linux.
I wanted to challenge myself further.

I was looking for a minimal linux distribution that is light on resources, and I found one called Metis Linux, which is based on Artix. The interesting part of metis is that it wasn't using a desktop environment, but a windows manager called dwm. At the time, metis linux had a minimal bash script installer via chroot. This took longer to setup, but I had a better understanding of what the setup involved rather than an automated GUI installer. I learned about how to partition the filesystem. I remembered I ran into some trouble was due to not running mkfs to make the filesystem! I was also planning to dual boot with windows, so I thought more about how to split my partitions. Each OS would have a dedicated portion of my SSD and HDD and a shared HDD portion that would as a generic media filesystem using exfat (since that was cross platform). This would contain my media files like music, pdfs, and other non OS specific files. I also set up a samba server to serve my media. I learned more about /etc/fstab and how that tells the OS what partitions to mount on linux.

This is what my laptop looks right now.

top bar is slstatus. wallpaper: Fantasy Lord of the Rings HD Wallpaper by Marcus Whinney

pywal is used to get the colorscheme from my wallpaper for my browser and terminal. left: neovim, top right: qutebrowser, bottom right: cava
From metis, I learned more about Artix and Arch (Since Artix is based on Arch). The cool thing about Artix is that it uses alternative init systems like openrc, runit, dinit, or S6 instead of SystemD. I'm using metis on both my laptop and desktop, so I will try Artix linux and another distribution in the future to mix things up. Linux From Scratch looks like a neat project to do as well.

Contribution to Open Source Projects

I discovered the moonlight client and sunshine server a few months ago. These are open source solutions to provide remote gaming/desktop capabilities with built in input and audio passthrough. I tried NoMachine, but I wasn't able to get audio to work. This looks like a known issue on arch. On sunshine, I didn't have to do any extra tweaking! This allowed me to game on my desktop pc without having to sit at my desk. This was especially helpful while watching my 2nd son. I was really impressed by the performance, I could stream my host's display at high resolutions and frame rates with low latency despite my desktop being in the basement using WiFi. I was getting some instability with WiFi, so I wanted to try connecting my desktop to the router via Ethernet. I decided to go with a headless solution because that gives me more flexibility on the placement of the desktop; I ended up moving my desktop upstairs closer to my router. I figured out a way to stream my hosts display headless by using Nvidia TwinView to create the virtual display. This means I don't need to buy any HDMI/DP dummy plugs. I wrote a Linux Guide for sunshine on how to set this up. If you have any feedback on this guide, let me know! I haven't tried this, but wolf is an interesting docker alternative to sunshine.

The sunshine project also got me curious in video encoding. I wanted to learn more about how to benchmark encoder performance and I discovered the encoder-benchmark. I noticed a small issue on linux that prevented the calculation of VMAF scores. Fortunately it was a simple fix. The filetime method didnt work on Linux, so I replaced the method with a cross platform one. I wasn't familiar with rust, so the concept of ownership confused me at first. I also noticed ffmpeg would hang forever when it times out. I added retry logic and timeout detection to ensure the permutor-cli runs to completion. Since there was no official linux support, I decided to contribute to the Arch User Reposititory (AUR). I have published a git and non-git version. The git version pulls the latest from main at the time of building. The not-git version pulls the current git tag.

Coding Projects

I posted a few things on Medium to experiment with dev blogging, but I don’t like the paywall system. I believe knowledge and ideas should be shared freely without inhibiting learning. Everything I post will be free and always remain so. I decided to try dev.to. I found out about this platform from this article. If you are curious about my previous posts you can see Mortgage App Update (React-Redux ElectonJS Desktop App) and How to Handle Opening and Closing Context Menus with a Custom React Hook. I most likely won't post more to medium, so I will stick with dev.to.

I always wanted to do my own project, but I had trouble finding a project I'm passionate in and remain motivated for the long term. I recently thought of an idea that combines nostalgia with some of my interests: gaming, coding, and machine learning into a project! I remembered my dad showed me the 1985 Bill Seiler's version of space War on his IBM PC/AT as a child.

You can play the emulator from your browser on internet archive
I remembered the controls are hard for the human player, so I'm curious if Reinforcement Learning agents can play the game better. First I will start building out the simulation/game until I have a minimal viable product to start training the agents. I will also recreate the original "AI" to measure baseline performance. I assume they are rules based and have access to internal state. The left player was defensive and mostly used phasers. The right player was offensive and mostly used photon torpedoes. My metrics of performance will be the game win rate. Once I get this going I will continue to make the simulation more complex and add more agents for Multi Agent Reinforcement Learning (MARL).

I'm planning to start a dev blog for this series to keep track of things I'm learning and discovering.

Here is a sneak peak of what I'm working on
basic player movement with zero gravity physics and screen wrap-around

I used OBS to capture my screen, shotcut to edit the video, and this command to create a gif (Shotcut also supports exporting to a gif, but it seems to take longer to process)

ffmpeg -i space-war-rl.mp4 \
  -r 30 -vf "scale=600:-1;" \
  -ss 00:00:15 -to 00:00:25 \
  space-war-rl-sm.gif