DEV Community: Red Rad

Kubernetes scaler for dedicated stateful servers

Red Rad — Tue, 25 Jan 2022 16:32:43 +0000

The Kubernetes autoscaling tries to add more pods in horizontal scaling and more resources (like memory or CPU) are added to the pod in vertical scaling. but while you are deploying a bunch of dedicated memory stateful servers (like game servers) you need to scale the node sizes instead of adding more pod or increasing pod resources.

I developed an open-source configurable k8s scaler server for the stateful dedicated servers inspired by Mark Mandel's (founder of Agones.dev) blog post. He had an awesome presentation on GDC about developing, deploying, and scaling dedicated game servers.

Kubescaler

The Kubescaler is a simple Kubernetes node scaler for dedicated servers. It runs the scale function in a loop and checks all nodes’ resources to determine when to scale up and down.

Slot

The slot is a pod that runs the game server and is limited to certain resources.

Scaler

The scaler considers a buffer of slots that must be available for new pods, so at the start, it compares the available slots (that is equal to node resource capacity divided to slot’s needed resources), if the available slot is smaller than the buffer size, it tries to scale up the node pool size and in other condition, it scales down the node pool size.

Scheduling nodes

The Kubernetes allows marking a node as unschedulable. It shows that is the controller can schedule a pod on the node or not. The scaler uses this option to help the extra node get empty faster.

Scaling up

Before starting to resize the node pool size, the scaler checks unschedulable nodes and starts to mark them as schedulable until the needed resource be supplied. if no unschedulable nodes are found or the needed resource didn’t supply, it resizes the node pool size to the needed size.

Scaling down

If the available slot is greater than the buffer size, the scaler starts to mark extra nodes as unschedulable, this option prevents the controller to schedule a new pod on the extra node. at the end, it checks the unschedulable nodes (extra nodes) for running pods, if no dedicated server pod is running and the node empty time passed the expiration, it deletes the node from node pool. Note that adding a new node to the node pool takes time, so an expiration time is considered for the empty node before deleting them, maybe a little while later the needed resource increased.

Selectors

The scaler lists the scalable nodes using node-selector option and uses the pod label selector to determine that is the node is empty of dedicated server pods or not.

Podwatcher

The scaler server watches the dedicated server pod creation or deletion events and triggers the scale function in addition to the time interval.

Cloud Provider

To resize the node pool size or delete a node we need to use the cloud API. The Digitalocean cloud provider is implemented in the Kubescaler. feel free to contribute and implement any other cloud provider and send the PR.
To use the Digitalocean API, you must set the cloud-provider-token and the cluster and node pool names.

The docker image exists here, also feel free to contribute to the repo:

https://github.com/theredrad/kubescaler

Pass secrets to Docker build to fetch private Github repositories

Red Rad — Tue, 28 Dec 2021 18:20:17 +0000

To fetch private repositories as dependencies in a Docker image build procedure, you must set the Github credentials.

The popular errors while an invalid credential is set:

fatal: could not read Username for ‘https://github.com': No such device or address

Git URL config

There are lots of posts on the internet to do this by setting your Github personal token in the git URL config, all of them are insecure because actually, they’re suggesting to pass your Github personal token to the Dockerfile as an argument and set it as the username of the Github repositories. remind that every command in Dockerfile, is printed on the build log and is accessible from the build history, so your private token is printed on the logs and ….

The SSH key method

You can also use an SSH key to download private repositories in the Docker build procedure. for this you need to generate a new SSH key and assign it to your Github account, then store the private key into the repository secrets and write it into a file in the Github action job step, then copy the file from the context directory to the .ssh directory of build container. By using this method, you are not in control of the SSH key scopes, the SSH key is your identity and has access to your whole account.

Using .netrc file

The .netrc file contains login and initialization information used by the auto-login process.

To use .netrc file, you must generate a new personal access token with private repositories related scopes, for example, if your private repositories exist under your account, therepo scope is enough, but if they exist under your organization or team account, theadmin:org scope is required.

Then store the generated personal access token in the repository secrets. Here I saved it with API_TOKEN_GITHUB name.

Create a Github action file:

name: Docker Image CI

on:
  push:
    branches: [ master ]

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - name: Store netrc file
        run: echo 'machine github.com login ${{ secrets.API_TOKEN_GITHUB }} password x-oauth-basic' > .netrc
      - name: Build the Docker image
        run: DOCKER_BUILDKIT=1 docker build --file Dockerfile --tag test-image:$(date +%s) --secret id=github,src=.netrc .

In this action, first, we store our secrets into the .netrc file, then pass the file as secret to the Docker build command.
Github action job hides the secrets in the logs, so it’s safe to use secrets in the run command.
After that, you must mount the secret file to root/.netrc in Dockerfile, (in this example we’re running Golang module download command after mounting the secret)

RUN --mount=type=secret,id=github,dst=/root/.netrc \
    go env -w GOPRIVATE=github.com/YOUR_USERNAME/* \
    && env GIT_TERMINAL_PROMPT=1 go mod download

Notice: You must run the go mod download or npm install commands right after mounting the secret file, actually in the same Docker Run command.

Using docker/build-push-action@v2

If you are using a Github action to build or push your image, you should pass the .netrc file like this:

# bluh bluh bluh 

- name: Store netrc file
  run: echo 'machine github.com login ${{ secrets.API_TOKEN_GITHUB }} password x-oauth-basic' > .netrc
- name: Push to GitHub Packages
  uses: docker/build-push-action@v2
  with:
    context: .
    push: true
    secret-files: |
      "github=./.netrc"

TL;DR

Don’t pass your secret to Docker build command as an argument, your secrets are printed in the build logs and history.
Using SSH key limits your control of the scopes.
You can store secrets in the .netrc file via the Github action and then pass it as a secret file to the Docker build command. (follow .netrc section)

How to use Lorem.space for generating random image placeholders with your local images

Red Rad — Mon, 15 Nov 2021 18:02:39 +0000

Lorem.space is an online API to generate categorized random image placeholders with custom sizes for your design layout.

What the hell is that?

Consider that you want to design a movie website or app, instead of downloading a bunch of movie covers, resizing them, and using them statically, you only set an URL (like https://api.lorem.space/image/movie?w=150&h=220) and each time a random movie cover will return.

To use API you can simply send a request to https://api.lorem.space/image/movie?w=150&h=220. This will return a random movie cover image in 150x220px dimensions. Some other categories like the game cover, music cover, store items, … are available and you can check https://lorem.space for more information.

How to run Lorem.space on your local machine

Lorem.space is an open-sourced program, you can download and run both frontend and backend apps on your local machine.

Run the server from binaries

Download the latest binary for your operating system: https://github.com/manasky/lorem-server/releases
Extract the downloaded file
Open Terminal (for macOS or Linux) and CMD (for windows) and navigate to the extracted folder.
Make sure the file is executable (grant permissions if needed)
Run the server with these commands:

# For macOS or Linux:
./lorem-server -host "127.0.0.1:8080" - dir "~/Pictures"
# This will run the server on port 8080 and serves images from ~/Pictures directory

# For Windows:
lorem-server.exe -host "127.0.0.1:8080" -dir "C:\Users\USER\Pictures"
# This will run the server on port 8080 and serves images from `C:\Users\USER\Pictures` directory

Run the server from source code

Lorem.space server is written in Golang and you can access the source code from here: https://github.com/manasky/lorem-server
To run the server from codes, follow these steps:

Install Go: https://golang.org/dl/
Clone the repository: git clone https://github.com/manasky/lorem-server.git
Go to the directory of source code, create a new directory named images and put your images (.JPEG format) here. Each directory in images is considered as a category. for example, you can create a directory named car and put all cars’ images there.
Open Terminal (for macOS and Linux) or CMD (for Windows), then cd to the directory of source codes and run: go run main.go
Now the server is running on 127.0.0.1:8080. Open http://127.0.0.1:8080/image in your browser and a random image from the images directory is returned.

Options

Some arguments are available to run the server:

host: The IP and port of the server host
dir: The address of the images directory
cache: Enables file caching to prevent resizing the image in the same request
cdn: CDN domain
min-width: minimum supported width size in px
max-width: maximum supported width size in px
min-height: minimum supported height size in px
max-height: maximum supported height size in px

Run the server with Docker

A built docker image of the Lorem server is available. You can pull the image by this command:
docker pull ghcr.io/manasky/lorem-server:latest
To run the docker image, you need to create a .env file containing the server configs:

HOST=0.0.0.0:8080
DIR=/app/images
MIN_WIDTH=20

Then in the directory of .env run this command in terminal:

docker run - env-file=./.env -p 8080:8090 -v ~/Pictures:/app/images ghcr.io/manasky/lorem-server:latest

— env-file sets the application environment variables path. Here .env file exists in the current path.
-p 8080:8080 sets up a port forward. the application container will listen to the 8080 port (set in the .env file as HOST). This flag maps the container’s port 8080 to port 8080 on the host (here, your localhost).
~/Pictures:/app/images sets up a bind-mount volume that links the directory /app/images from inside the app container to the directory ~/Pictures on the host machine (here, your system). The app image directory is set in the .env file as /app/images, so here the ~/Pictures is mounted to that.

After container running, the app will scan /app/images which is mounted to the directory ~/Pictures of your system & serves the HTTP server on 127.0.0.1:8080. Visit 127.0.0.1:8080/image for start.

Contribute

Feels free to contribute to the repository or file an issue for feedback or bug reporting in Github.

UDP Socket manager for go server-side

Red Rad — Tue, 05 Oct 2021 13:04:55 +0000

I'm working on a prototype online multiplayer game. As you know an online multiplayer game has a server-side process that manages the game state & simulates the world based on the users' inputs.

The game is fast-paced, so I used the UDP protocol to create a virtual channel between the client & the server. The UDP is a stateless protocol, which means that the server & client are not aware of a connection, they only exchange packets. Also, there is no guarantee that the packets are received in order or receive at all.

The Security

To manage the network connections there is a major challenge, security! this challenge has two sides, the first is related to user authentication & session management & the second is secure the content of the packets around the internet.

Session management

Each user input that is received in the server, must be authorized.

Encryption

To secure the content of packets around the internet we need to use encryption.

So I decided to implement a simple general-purpose package for Golang to manage the incoming packets & provide security for the virtual connection between the server & the client.

The `udpsocket` package

I made this package to make a virtual stateful connection between the client & server using the UDP protocol for a Golang game server (as you know the UDP protocol is stateless, packets may not arrive in order & there is no ACK). check it out on Github

The udpsocket supports a mimic of DTLS handshake, cryptography, session management & authentication. It's responsible to make a secure channel between the client & server on UDP (handshake), authenticate & manage them, decrypt & encrypt the data & provide an API to send or broadcast data to the clients & listen to them.

Custom Implementations

Feels free to:

implement your auth client to authorize the users' tokens
implement your preferred symmetric & asymmetric encryption algorithms (the RSA & AES encryption are supported as default)
implement your preferred encoding package. (the protobuf is supported as default)

Basic Usage

package main

import (
    "context"
    "log"
    "net"
    "fmt"
    "crypto/rsa"

    "demo/auth"
    "demo/encoding"

    "github.com/theredrad/udpsocket"
    "github.com/theredrad/udpsocket/crypto"
)

var (
    pk *rsa.PrivateKey

    udpServerIP   = "127.0.0.1"
    udpServerPort = "7009"

    defaultRSAPrivateKeySize = 2048
)

func main() {
    f, err := auth.NewFirebaseClient(context.Background(), "firebase-config.json") // firebase implementation of auth client to validate firebase-issued tokens
    if err != nil {
        panic(err)
    }

    udpAddr, err = net.ResolveUDPAddr("udp", fmt.Sprintf("%s:%s", udpServerIP, udpServerPort))
    if err != nil {
        panic(err)
    }

    udpConn, err = net.ListenUDP("udp", udpAddr)
    if err != nil {
        panic(err)
    }
    defer udpConn.Close()


    pk, err = crypto.GenerateRSAKey(defaultRSAPrivateKeySize)
    if err != nil {
        panic(err)
    }

    r := crypto.NewRSAFromPK(pk) // creating a new instance of the RSA implementation
    if err != nil {
        panic(err)
    }

    a := crypto.NewAES(crypto.AES_CBC) // creating a new instance of the AES implementation

    t := &encoding.MessagePack{} // an implementation of msgpack for the Transcoder
    s, err := udpsocket.NewServer(udpConn, &udpsocket.Config{
        AuthClient: f,
        Transcoder: t,
        SymmCrypto: a,
        AsymmCrypto: r,
        ReadBufferSize: 2048,
        MinimumPayloadSize: 4,
        ProtocolVersionMajor: 0,
        ProtocolVersionMinor: 1,
    })
    if err != nil {
        panic(err)
    }
    s.SetHandler(incomingHandler)

    go func() { // handling the server errors
        for {
            uerr := <- s.Errors
            if uerr != nil {
                log.Printf("errors on udp server: %s\n", uerr.Error())
            }
        }
    }()

    go s.Serve() // start to run the server, listen to incoming records

  // TODO: need to serve the public key on HTTPS (TLS) to secure the download for the client
}

func incomingHandler(id string, t byte, p []byte) {
    // handle the incoming
}

Dig deeper, How it works

Record

Each message from the client is a Record. The record has a format to parse & decryption.

 1   0   1   1 0 2 52 91 253 115 22 78 39 28 5 192 47 211...
|-| |-| |-|  |------------------------------------------|
 a   b   c                        d

a: record type
b: record protocol major version
c: record protocol minor version
d: record body

Record Types

The first byte of record is the type & indicates how to parse the record. supported reserved types:

ClientHello : 1
HelloVerify: 2
ServerHello: 3
Ping: 4
Pong: 5

Handshake

The handshake process is made base on a mimic of DTLS protocol, the client sends ClientHello record, this record contains a random bytes & AES key & encrypted by the server public key, which is needed to download on TLS, then the server generates a random cookie based on the client parameters, encrypts it with the client AES key (which is received by ClientHello & sends it as the HelloVerify record. The client decrypts the record & repeats the ClientHello message with the cookie, the record is needed to be encrypted with the server public key, then append the encrypted user token (with AES) to the record body. the server will registers the client after cookie verification & authenticates the user token & then returns a ServerHello record contains a random secret session ID. The handshake process is done here.

      Client                                   Server
      ------                                   ------
      ClientHello           ------>

                            <-----         HelloVerifyRequest
                                           (contains cookie)

      ClientHello           ------>
 (with cookie & token)

                            <-----           ServerHello
                                         (contains session ID)

Encryption

The Server uses both symmetric & asymmetric encryptions to communicate with the client.

The ClientHello record (for the server) contains a secure 256 bit AES key & encrypted by the server public key, so the server could decrypt it with the private key.

The HelloVerify record (for the client) encrypts with the client AES key (which was decrypted before).
If user authentication is required, the client must send the user token with the ClientHello (a record which contains cookie too), but the asymmetric encryption has a limitation on size. for example, the RSA only is able to encrypt data to a maximum amount equal to the key size (e.g. 2048 bits = 256 bytes) & the user token size could be more, so the user token must be encrypted by the client AES, then the server could decrypt it after validation of ClientHello record.
The handshake record structure is a little different because of using hybrid encryption. the two bytes after the protocol version bytes indicates the size of the handshake body which is encrypted by the server public key. the handshake body size is passing because of the key size, the encrypted body size depends on the RSA key size.

 1   0   1   1   0   2 52 91 253 115 22 78 ... 4 22 64 91 195 37 225
|-| |-| |-| |-| |-| |---------------------|   |--------------------|
 a   b   c   d   e            f                         g

a: record type (1 => handshake)
b: record protocol major version (0 => 0.1v)
c: record protocol minor version (1 => 0.1v)
d: handshake body size ([1] 0 => 256 bytes, key size: 2048 bits) (first digit number in base of 256)
e: handshake body size (1 [0] => 256 bytes, key size: 2048 bits) (second digit number in base of 256)
f: handshake body which is encrypted by the server public key & contains the client AES key
g: user token which is encrypted by the client AES key size

Possible issues

If there were any possible issues or needs, please put comments or collaborate to make it better. Repository on the Github

DEV Community: Red Rad

Kubernetes scaler for dedicated stateful servers

Kubescaler

Slot

Scaler

Scheduling nodes

Scaling up

Scaling down

Selectors

Podwatcher

Cloud Provider

Pass secrets to Docker build to fetch private Github repositories

Git URL config

The SSH key method

Using .netrc file

Using docker/build-push-action@v2

TL;DR

How to use Lorem.space for generating random image placeholders with your local images

What the hell is that?

How to run Lorem.space on your local machine

Run the server from binaries

Run the server from source code

Options

Run the server with Docker

Contribute

UDP Socket manager for go server-side

The Security

Session management

Encryption

The udpsocket package

Custom Implementations

Basic Usage

Dig deeper, How it works

Record

Record Types

Handshake

Encryption

Possible issues

The `udpsocket` package