DEV Community: Amir Eldor

Python pytest testing with SQLAlchemy for my game

Amir Eldor — Fri, 04 Jun 2021 14:59:18 +0000

In this development blog for my online single/coop 4X space game post I will write about my Python pytest testing with a database environment.

My game has to track progress of things. Be it research progress, build queues, amount of resources, etc. I use a table that I call progress for that, at least for the purpose of building the Test Game in the process of realizing how I will do things in the Real Game.

This table would be updated with a periodic task running in Celery, and as I wrote the task's function I realized I'm gonna need some automated tests. It would be painful to code such task, run a worker locally, monitor the database for changes, fix a bug, do it all over again, and so on. Not to mention the pains of refactoring which will surely happen as I realize what my data structures should better be like.

So I had a few funny rounds of wresting with pytest and a database session fixture and I eventually came up with something like the following (from my current code):

# NOTE: I am not very experienced with SQL/ORM/RDBS so I may be doing
# stupid things. Please comment below to let me know if I can do something
# better or if I made you pull your hair with disgust

# TODO: probably be function scope, then make create_ and drop_all in another fixture
@fixture(scope="session")
def db_session():
    Base.metadata.create_all(engine)
    session = Session()
    yield session
    Base.metadata.drop_all(engine)
    session.close()

And an example for a test that uses this:

# NOTE: I am not very experienced with SQL/ORM/RDBS so I may be doing
# stupid things. Please comment below to let me know if I can do something
# better or if I made you pull your hair with disgust

# TODO: move somewhere more common
@fixture
def game(db_session):
    game = GameMeta(name="testgame", seed=0)
    db_session.add(game)
    db_session.commit()
    yield game
    db_session.delete(game)
    db_session.commit()


def test_advance_all_games_progress(db_session, game):
    progress1 = Progress(progress=0.0, game=game)
    progress2 = Progress(progress=0.0, game=game)
    db_session.add(progress1)
    db_session.add(progress2)
    db_session.commit()

    advance_all_games_progress()

    stmt = select(Progress)
    result = db_session.execute(stmt)
    for progress in result.scalars():
        assert progress.progress > 0.0

    db_session.delete(progress1)
    db_session.delete(progress2)
    db_session.commit()


def test_progress_increment(db_session, game):
    progress = Progress(
        progress=1.0, game=game, increment_min=10.0, increment_max=100.0
    )
    db_session.add(progress)
    db_session.commit()

    advance_all_games_progress()
    progress = db_session.get(Progress, progress.id)
    assert progress.progress >= 10.0

    db_session.delete(progress)
    db_session.commit()


def test_progress_cap(db_session, game):
    progress = Progress(
        progress=1.0, cap=2.0, game=game, increment_min=10.0, increment_max=100.0
    )
    db_session.add(progress)
    db_session.commit()

    advance_all_games_progress()
    progress = db_session.get(Progress, progress.id)
    assert progress.progress > 1.0
    assert progress.progress <= 2.0

    db_session.delete(progress)
    db_session.commit()

It seems that SQLALchemy sessions support some sort of session-inside-a-session or nested-sessions that I should look into. The function advance_all_games_progress opens a session inside itself, and it is called from within the db_session fixture.

I initially tried doing every operation (creating mock rows and objects, destroying them later on, for example) with a db_session of its own. It was a disaster. Once you create an ORM-based object in a session it is tied to it, and once the session ends you end up with something that can't realize its lazy-loaded relationships and the likes. There's a method called expunge that you can do for these kind of objects to "untie" them from a session, but then I wasn't sure if I should re-tie them to a new session for later usage (imagine the game fixture). I did all kinds of random programming guesses and some shots in the dark while Googling, but with a simple db_session fixture and a lot of db_session.commit I got stuff to work and I think I'm even not doing anything strange. Please fix me if I am totally wrong, thanks :)

Now there's the problem of actually having a test database to test things against. Using Docker, I started a local Postgres server and created a test database for game testing. It looks like this (from my README):

docker run -d --name postgres -e POSTGRES_PASSWORD=password -p5432:5432 postgres:alpine
docker run --rm -it --network host postgres:alpine psql postgres://postgres:password@localhost:5432/
# or if you have the postgres-client package installed:
#   psql postgres://postgres:password@localhost:5432
CREATE DATABASE gametest;

I mentioned a README. Even being closed-source (for now?) I find it important to include instructions for my future self for doing things with my projects.

My .env.test file has this entry:

DATABASE_URL=postgres://postgres:password@localhost:5432/gametest

And somewhere in my code which is FastAPI-based and thus uses Starlette:

# config.py

TESTING = environ.get("TESTING")

config = Config(".env" if not TESTING else ".env.test")

...

DATABASE_URL = config("DATABASE_URL", default=None)

Lastly, my conftest.py (for pytest) ensures the TESTING env var is present:

from starlette.config import environ

environ["TESTING"] = "True"

Now that I think about this, it's a bit strange that code that is not supposed to run for the Heroku web worker is using starlette configs. That is, my "workers" which are supposed to run tasks for the game, do include reference to starlette-based configs. Oh well. Things work and for now my focus is really on actually making a game so I must overlook these kind of things, which might not be issues at all.

Early Heroku and App Platform infrastructure, with some thoughts about letting you know when your colony ship has been built

Amir Eldor — Fri, 28 May 2021 12:24:22 +0000

Hello. In this part of my development blog I will describe the infrastructure I chose for the early development of my online persistent universe single/coop players space game, AGS9. I will try to describe the reasons for my choices and describe some technical stuff. This is what I'm going to implement next in my development process.

To ease deployment and not "fiddle around" with servers and infrastructure, I chose to run my game server under Heroku. Instead of choosing a real-time process that "runs in the background all the time" and calculates the next "tick" of the universe in some sort of gameloop, possibly using an ECS (Entity Component System) for tracking all the things that should be going on, I decided to go the old "classical web-app" way and have a relational database and a web server.

Even though the game "seems" real time, it doesn't have to be as responsive as a first-person-shooter, and I can have the network latency of using a relational database to answer questions from the client like: "what planets are in this solar system?" or "how many resources do I have?". The client can also ask the web server, or more appropriately game server, requests like "build me a nice colony ship". In this case, I intend to write a "build entry" in some table and then do something a bit strange.

The something that is a bit strange is this. I have a task/event system, in this case a Celery worker with a persistent Redis backend (also on Heroku), and I write a delayed task, 5 hours from now that will write to some queue (probably Redis again) that a colony ship was built. I need to check the database before I publish this event, because the user might have chosen to cancel the build, and I don't have an easy or reliable way to remove the "ship built" event from the queue.

The game server, i.e. the web process on Heroku, listens to the queue I mentioned. Upon receiving the successful "ship built" event, I notify all connected WebSocket clients of the event so they can shows a nice message to the player. The ship is also added to the fleet table for the player (probably by the worker though, not the web process).

This is a bit strange, but I prefer to have an event-based system instead of some sort of "DB-queue", that queries a "next builds" table for rows with an "end time" that is before Now, and acts accordingly. I would hate having such loop run in some 1-second cron, possibly not doing a lot each iteration.

The persistence of the Redis server should keep the delayed tasks after a restart or failure of Redis, so I hope I will be fine in this case.

I do need to read a bit and think of a case where I'll have to scale the Redis servers while keeping persistence between them. That's an interesting point.

A few words on the client side.

Currently, in the current development stage, I host two static websites on the DigitalOcean App Platform. The first is a statically-exported Next JS app that authenticates the player with Auth0. The other website is a Phaser-based "game-only" app that connects to my Heroku server with a WebSocket. There's a small "handshake" after the WebSocket connection is established, where the Auth0 access token is passed to the server for validation.

The reason for separating the two, is that the first site is built quickly to just give the most basic authentication and managing games functionality, and after the game is more mature I could replace this "portal" or "lobby website" with a well-polished drop-in replacement I will build, without changing the code for the Phaser game client.

I had to come up with a way to pass over the token from the Next JS site to the Phaser app. I came up with having a page under the Next app that loads the other game app in a full-viewport iframe and uses postMessage to pass the token. Passing the token via a query parameter for example is bad for various reasons, and I didn't want to keep the token in a global window.something variable, to defend the token (over-engineered security :)).

These websites are deployed with the App Platform of DigitalOcean, which alongside Heroku gives me some sort of rudimentary CI/CD system that lets me focus on app-development and not deployment and cloud infrastructure.

A quick tip with Next JS static sites and the App Platform is to add trailingSlash: true to your next config, so a route like mydomain.com/play is not a 404 with the App Platform's current routing. The exported play.html page is not accessible on such route, but if you have a play/index.html it works. See this: https://www.digitalocean.com/community/questions/nextjs-export-redirect-path-without-extension-to-path-with-html?answer=65145

I hope this is not too boring. In the next part I intend to write about the first 'test game' I am going to develop and how I built it.

Introducing a Development Blog about my Online Persistent-Universe Game

Amir Eldor — Fri, 28 May 2021 12:24:01 +0000

Hello. AGS9 is a game I'm developing. It will be online, single player or cooperative private games with a persistent universe that keeps on running even when you are logged out. The theme will be space exploration and survival, with stars to colonize and space dragons to defend against. The intention is to create a 4X/resource-balancing game that takes inspiration from the gameplay of Frostpunk and Master of Orion (of course), with the anticipation of classic web games like Tribal Wars for something that will happen in a few hours or days from now.

AGS9 is not intended to be PVP at all, at least not in my original design. I want to create a game that people with a full-time job can play and excel at, without 14 year old kids farming them all the time.

I decided to write down a development blog about my progress and the challenges I encounter and overcome. These writings come not from the first line of code I write, but after several iterations of different game and server architectures I played around with and learned what suits best for the game at this stage of development.

For example, I decided against running multiple microservices deployed with Kubernetes, either with a Stateful Set of game servers that host multiple games, or a crazy idea of a Pod per game instance. I examined a solution of a Godot client/server architecture, but having HTML5 as a target platform and reading in Godot's official documentation that HTML5 support is not that great, I dropped that project too. That idea included either multiple games per a Godot server instance, or a Pod for a single Godot server that runs a single game for a player. The latter case is interesting because it lets me develop the game regardless of infrastructure or any authentication logic, multiple games on a single server, etc, and I only need some API Gateway Server for Kubernetes to have my players authenticated and directed to the individual game Pods.

But I don't do any of that. It's an overkill in this early stage of the game development.

Did I mention there's no game yet? Only infrastructure? :)

Today, at the time of this writing, I understood that as a developer I tend towards building things with cool technologies instead of actually making a Product, in this case a game. I do know however, that I want to have a working end-to-end system, with login, user management, and a "skeleton game" online and running, before I iterate on the game itself.

When I know everything "works" from the client side login, up to the server which communicates with the client through WebSockets with pushes of "space dragons attack you" or "your ship is ready after 7 hours of building", I could start focus on the game design itself, and inviting users slowly so I can get early feedback and develop with the community accordingly.

It's worth mentioning that I intend to run the game under a very cheap subscription plan, just to keep the servers running and not to get rich or something like that.

Trying to make these posts not too long, I will write about the current "early development architecture" I chose in the next part of this blog.

A Godot server + client game deployed on DigitalOcean Apps

Amir Eldor — Fri, 16 Oct 2020 23:30:51 +0000

I'm going to tell you how I built a Godot client/server game deployed on DigitalOcean Apps. Well, it's more of an experiment than a game at the moment, but I thought it's worthwhile sharing as Godot server resources on the net seem to be scarce, let alone deployment of such.

The game will be an online single-player persistent-universe game. The server should support multiple players playing on the same server, each in their sandbox.

Initially choosing Python and Phaser3 for development, my partner humbly asked me why don't we use an existing game engine, and I did not have a decent answer to give. Recalling a famous quote from somewhere around the web: "make games, not engines." I decided I have to try at least making a proof-of-concept for this kind of game, using a game engine. Enter Godot. Why not Unity? See further down this article.

Godot "universal" server/client Space Game

By leveraging the Godot's High-Level multiplayer networking API I can have the engine handle tasks such as syncing state between the server and client as well as issuing bidirectional commands between the peers. The client might send a "get star system data" or "fly a spaceship to planet X" and the server might report a random "space dragon attack" event or any other game-related data and logic.

A requirement for the game is running on a browser, so I chose the WebSocketMultiplayerPeer class in the GDScript code.

Let's connect things

We are developers, I'll talk less and show more code.

func _ready():
    var peer: WebSocketMultiplayerPeer
    if OS.has_feature("Server") or "--server" in OS.get_cmdline_args():
        peer = WebSocketServer.new()
        peer.active_relay = false  # see note below
        peer.listen(4321, [], true)
    else:
        var server_url = 'ws://localhost:4321'
        if OS.has_feature('release'):
            server_url = "wss://this.will.be.your.digitalocean.apps.thing.soon/server"
        peer = WebSocketClient.new()
        peer.connect_to_url(server_url, [], true, [])

    if peer == null:
        OS.exit_code = 1
        print("Failed initializing peer :( so can't network things")
        get_tree().quit()

    get_tree().network_peer = peer
    if get_tree().is_network_server():
        print("I am the mighty server")
        var ok
        ok = get_tree().connect("network_peer_connected", self, "_peer_connected")
        assert(ok == OK)
        ok = get_tree().connect("network_peer_disconnected", self, "_peer_disconnected")
        assert(ok == OK)
    else:
        print("I am an unworthy client")
        var ok
        ok = get_tree().connect("connected_to_server", self, "_connected_to_server")
        assert(ok == OK)
        ok = get_tree().connect("connection_failed", self, "_connection_failed")
        assert(ok == OK)

The peer.active_relay = false command is so the server does not broadcast the connection of each peer to all the other connected peers (thanks LordDaniel09).

Forgive me for all the OK assertions, I am rather new to Godot and a combination of not knowing what I'm doing and compiler warnings made me do that.

This code will initialize a server or connect to one depending on some feature or command-line argument. To run the server I just run this from the project's folder:

godot-server

You need the server build of Godot for that. This is what we'll deploy eventually with the Dockerfile on DigitalOcean Apps.

Running the client is simply running godot or running the game from the editor. I also obviously want an HTML5 build so I exported the game rather early and made sure all this Websocket madness works.

Creating a new game or joining an existing one

On the client-side, when we connect to the server, we create a Game Scene locally and then ask the server (peer id 1) to create game 123 for us or let us join the one that is already running. Authentication, unique Game IDs, and game->player association is still something I'm thinking about. More on that soon.

We need the local scene so the hierarchy on the server which will soon create a similar node will be the same for the two peers. This makes Godot happy when doing the high-level multiplayer magic, or so I understand.

func _connected_to_server():
    print("Connected to server!")
    # TODO: Should explicitly ask to make a new game or join
    var game_id = 123  # get from somewhere
    var game = load("res://Game.tscn").instance()
    game.set_name(str(game_id))
    $Games.add_child(game)
    rpc_id(1, "_join_game_or_new", 123)

The hierarchy will look something like this:

And on the server:

remote func _join_game_or_new(game_id: int):
    var peer_id = get_tree().get_rpc_sender_id()
    print("%d wants to join game %d" % [peer_id, game_id])
    # TODO: make sure peer allowed to join this game

    var game_node_path = "./Games/%d" % game_id
    if has_node(game_node_path):
        pass # What now?
    else:
        print("Creating game %d" % game_id)
        var game = load("res://Game.tscn").instance()
        game.set_name(game_id)
        $Games.add_child(game)

So at this point, both the client and server have "agreed" on the hierarchy and the game can start (or resume as we'll see).

The Game scene contains a Solar System scene. In the future, we might have a Galaxy scene or Planet or any other screen that we might need.

Inside the Solar System scene, we start having interesting fun. Let's look at the _ready() of that scene.

func _ready():
    if is_network_master():
        print("Randomizing a solar system")
        var rng = RandomNumberGenerator.new()
        rng.randomize()
        # Following is very specific mock-ish logic code just for the sake of having something on the screen
        var radius = 200
        for _index in range(rng.randi_range(2, 10)):
            var planet = load('res://Planet.tscn').instance()
            planet.set_planet_kind(rng.randi()%20 + 1)
            var pos = Vector2(radius, 0)
            pos = pos.rotated(rng.randf_range(0, 2*PI))
            planet.position = pos
            # this is stupid:
            planet_angle_inc.append(rng.randf_range(min_angle_inc, max_angle_inc))
            radius += 230
            $Planets.add_child(planet)
    else: 
        call_deferred('rpc_id', 1, "get_planets")

For the client, which is not the network master at any point for any node (besides GUI, which is not networked either way), I call get_planets with call_deferred. As you'll see in the implementation of get_planets, the scene should exist for the client so _ready() should have been run already when this call is made. Well, that's what I tell myself, as without call_deferred I had "Failed to get path from RPC" errors.

remote func get_planets():
    var peer_id = get_tree().get_rpc_sender_id()
    print("Sending planets to peer %d" % peer_id)

    # not sure if this is the best way to serialize things
    # If you have Godot magic I'm unaware of please let me know
    var serialized = {"planets": []}

    var planets = $Planets.get_children()

    for index in range(len(planets)):
        serialized["planets"].append(
            {
                "position": [planets[index].position.x, planets[index].position.y],
                "angle_inc": planet_angle_inc[index],
                "kind": planets[index].get_planet_kind(),
                "rotation": planets[index].get_rotation(),
                "rotation_speed": planets[index].rotation_speed,
            }
        )

    rpc_id(peer_id, "set_planets", serialized)

The planets are rotating around the star. As part of the POC, I wanted to make sure I get a consistent state between the client and server, even when I refresh or restart the client. So if for example a planet just passed "3 o'clock" and I'd reconnect briefly after, I'll see that planet a bit more towards "4 o'clock" and not in a completely new random position.

It's worth noting that the Solar System scene is initialized on the server once, and I intend to keep it alive as long as the game lives. This will be needed for the whole universe simulation to keep running in the background. The client however will delete and create new Solar System scenes as they go through the systems in the galaxy. There's no need for a client to know all the mysteries of the universe and it would be a waste of resources (think mobile browsers).

remote func set_planets(serialized):
    for serial in serialized["planets"]:
        var planet = load("res://Planet.tscn").instance()
        planet.set_planet_kind(serial["kind"])
        var pos = Vector2(serial["position"][0], serial["position"][1])
        planet.position = pos
        planet.rotate(serial["rotation"])
        planet.rotation_speed = serial["rotation_speed"]
        planet_angle_inc.append(serial["angle_inc"])
        $Planets.add_child(planet)

    first_sync_done = true

At this point, the server and client should have the same state. And the sweet part is that because we run the game on the same engine, then the simulation of the heavenly figures will be in sync without a need to continuously update the positions of the planets.

func _physics_process(delta):
    var planets = $Planets.get_children()
    for index in range(len(planets)):
        var planet = planets[index]
        var pos = planet.position
        pos = planet.position.rotated(planet_angle_inc[index] * delta)
        planet.position = pos

I will however add additional get_planets call from the client to the server every, say, 10 seconds, just to be sure we are really in sync.

Deploying the server and a static HTML5 website to DigitalOcean Apps

So the only viable option for running this on the DigitalOcean Apps PaaS is with a Dockerfile. Actually two Dockerfiles. One for the main server and the other for building the HTML5 build of the game and hosting that on a static website in the platform.

Searching DockerHub, I found the barichello/godot-ci repo which was a fantastic replacement to my flaky Docerfile build. Basically what my Dockerfile did and the repo's one do is download a copy of godot-headless, which can build things e.g. the HTML5 build.

Let's start with what I called Dockerfile.html5 as it's a bit easier.

FROM barichello/godot-ci

WORKDIR /game
COPY . ./src/

RUN mkdir build && godot --path ./src/ --export HTML5 /game/build/index.html && rm -r src/

We copy the project, build things to some build folder, and then remove the code to keep the image leaner. The build is standalone and does not require the code at all even for HTML5.

For the game server, we also need to download godot-server so the Dockerfile would look something like this:

FROM barichello/godot-ci

RUN wget https://downloads.tuxfamily.org/godotengine/3.2.3/Godot_v3.2.3-stable_linux_server.64.zip && \
    unzip Godot_v3.2.3-stable_linux_server.64.zip && \
    mv ./Godot_v3.2.3-stable_linux_server.64 /opt/godot-server

WORKDIR /game

COPY . ./src
RUN godot --path ./src --export-pack Linux /game/game.pck && rm -r ./src

EXPOSE 4321
CMD [ "/opt/godot-server", "--main-pack", "/game/game.pck" ]

It took a bit of trial and error to get the build paths right, also for DigitalOcean Apps to pick the static website folder, but I managed eventually.

Let's see some DigitalOcean Specifics

Deploying the server was pretty easy. For the static website, I needed to use doctl to edit the spec for my App so I can provide a Dockerfile path for the static build.

The commands are more or less like:

doctl apps list
# take note of your app id
doctl apps spec get YOUR_APP_ID > do-apps-spec.yaml
# ...or any other filename, not sure how DO want you to call it

Then you'd edit the static website's section and add a Dockerfile_path and the now-required output_dir:

domains:
  - domain: this.is.stil.unpublished.com
    type: PRIMARY
name: your-repo-name
region: fra
services:
  - dockerfile_path: Dockerfile
    github:
      branch: live
      deploy_on_push: true
      repo: me/repo-name
    http_port: 4321
    instance_count: 1
    instance_size_slug: basic-xxs
    name: nice-server
    routes:
      - path: /server
static_sites:
  - envs:
      - key: SERVER_URL
        scope: BUILD_TIME
        value: wss://the-digital-ocean-link-for-the-server/server
    github:
      branch: live
      deploy_on_push: true
      repo: me/repo-name
    name: web
    routes:
      - path: /web
    dockerfile_path: Dockerfile.html5
    output_dir: /game/build

Update the App's spec:

doctl apps update $APPS_ID --spec do-apps-spec.yaml

And I think that did it!

It was a bit painful, but eventually, I got a game up. When I refresh the browser, I get the planets' state as they are on the server in my little persistent universe, which keeps on turning while I am not actively connected to it.

Show me the code

I showed you.

J/K.

I think the whole code as a "clone and run" repository is redundant. I gave you the ideas and snippets and you'll educate yourself better if you do some more hands-on on this one.

Authentication and Game IDs

I lean towards an external auth service such as Auth0 to generate some lovely JWTs for me. Then I would need a way for Godot to validate these JWTs.

When a client connects to the server, I could either use a C# or C++ module that handles JWT validation as suggested here or maybe use a cloud function of some sorts to validate the token before letting the peer know the secrets of the universe.

Why not Unity?

I chose Godot for 2 reasons.

One, it's open-source and I thought that if I manage to get this game somewhere then a "made with Godot" banner would be very nice for both projects.

Second, Unity's network stack is being re-written at the moment. I did not want to play around with a deprecated API. One could say I can use Mirror for Unity and they would be right, however, when trying this a few months ago I didn't get the HTML5 build to work properly and it's something I want for my game.