DEV Community: Alexander Demin

Compiler for "Easy" language from "Etudes for Programmers" book (1978)

Alexander Demin — Tue, 14 Oct 2025 16:56:59 +0000

EASY language compiler

The repository contains a compiler for the educational programming language called Easy, as described in the book Etudes for Programmers (1978) by Charles Wetherell.

NOTE: The compiler is also educational only. The compiler was written from scratch, and I had no compiler-writing experience before. My purpose was to learn compiler implementations and runtimes.

EASY language intro

The program below is from the book. It is intended to demonstrate the gist of the EASY language. This code was my very first milestone when developing the compiler:

Here is the source of the program and also the output of the compiler.

Implementation

The compiler is implemented in Typescript. The compiler emits C code, which Clang or GCC then compiles into the native binary. The runtime.c file is a bare minimum runtime support. In the x subfolders of the tests, there are test.c files, representing the output of the compiler.

The compiler requires a Javascript runtime (Node 24+ or Bun) and Clang 17+.

To compile and run a program in Easy, you run "easyc run filename.easy", for example:

easyc run life.easy

or step by step:

node easyc.ts life.easy && cc life.c -o test -I . && ./test

easyc.ts compiles life.easy to life.c, and then Clang compiles easy.c to the executable.

The Easy language syntax is fully supported, according to the book. However, there are a few points worth mentioning.

EXTERNAL subroutines and NAME aliases are allowed but not supported semantically.

Multiple PROGRAM segments are not supported, and the program should have only one PROGRAM segment, which becomes its entry point. The identifiers (type, variables, subroutines) from the PROGRAM segment are hoisted to the global namespace and visible in all parts of the Easy program.

According to the book, Easy is a copy semantics language. It means that copying a primitive type, a structure, or an array always makes a full, deep copy. The subroutine arguments and the function return value are also copied deeply to provide the value semantics.

The only exception to the value semantics is the array, whose size is not known at compile time. Such arrays are allocated dynamically, but the copy operation (an assignment, passing as a subroutine argument, or returning as a result from the function) performs a shallow copy (a runtime limitation). Also, the arrays in the top-level PROGRAM segment must have a compile-time known size.

The compiler uses C struct to implement compound types, such as strings, arrays, and structures.

The compiler implements a handwritten recursive descent parser. However, there is a second experimental PEG parser (peg.ts with easy.peg grammar file), but this parser is not used to generate code. The PEG parser only runs in testing by test.ts. Maybe in the future, PEG will replace the basic recursive descent parser.

Compiler test pipeline

The compiler project has a test pipeline, test.ts, running a set of tests from the tests folder. Some of the tests implement well-known programs, such as the Brainfuck interpreter, Conway's Game of Life, Rule 110 automaton, Quine (a program that prints its own source code), Mastermind (utilising Knuth's Minimax algorithm), FizzBuzz, Eratosthenes Sieve (from the book).

The "tests" folder contains compiler tests. Each subfolder is an individual test. Inside each test folder, there is a file named test.easy. In the "x" subfolder of the test, there are expected ("golden") files that are expected to be generated by the test.

There are several types of expected files. The main ones are test.c and test.output. The first is the compiler output. This file is then compiled to test.exe by the C compiler. The test.output is the console output of the test.exe execution. If a test requires console input (such as sieve), a test.input file is available.

Additionally, a test may have test.tokens (lexer tokens), test.s (symbol table), test.json (AST), and test.peg.json (AST from PEG). These files are optional and not all tests use them.

Testing and running examples

Prerequisites

JavaScript runtime: bun or node (24+) (remember - the compiler is written in TypeScript)
clang 17+ (to compile the Easy compiler output to a native binary)
just (make alternative)
docker (optional, if we want to run tests in an isolated container)

Currently, the compiler test pipeline runs either locally by just test-compiler or by just docker-test.

NOTE: Running tests in the Linux container helps because the Clang memory sanitiser is more capable in Linux, rather than on macOS.

Examples

Get a feel for the compiler, and run just life to play the Convey's Game of Life in the console.

Other examples:

just run bf - to run a Brainfuck program printing "EASY!".

just run fizzbuzz - FizzBuzz

just run quine - the program which prints its source (an etude from the book)

just run mastermind - to play Mastermind, so the computer will guess your code using the Knuth's minimax algorithm for Mastermind (also an etude from the book about)

just run sieve - to run the Eratosthenes Sieve (from the book) - enter a maximum number, and the program will find all primes up to this number.

just run rule_110 - to run Rule 110 automaton.

Compiler internals overview

As mentioned above, the compiler utilises a manual recursive descent parser, as specified in the Easy grammar specification from the book.

Each AST node can emit C code in either c() or v() functions. v() function is used by expression nodes. The v() function emits C code and also the name of the variable with the result of the expression.

Outro

As I mentioned at the beginning, this is a purely educational project where I wanted to learn how to write compilers. I never implemented computers before, and implementing one was an interesting challenge.

easy-cot.yaml - syntax highlighting for Easy for the macOS "cot" editor
https://github.com/begoon/easy-vscode - a VSCode extension for Easy syntax highlighting
https://github.com/begoon/easy-lsp - a prototype of the VSCode LSP for Easy (proof of concept)

"Hello World" in HTTP way

Alexander Demin — Sun, 22 Dec 2024 15:14:03 +0000

Problem statement

Implement a "Hello World" application in any programming language of your liking.

The application is a CLI application. It first launches an HTTP server on "localhost:8000." The server should respond with a text response (content-type "text/plain") and response code 200 on any route. When the HTTP service is ready, the application sends a GET request to its HTTP server, reads the response ("Hello World"), and prints the response to the standard output. Finally, the application shuts down the HTTP server and exists.

The application can be tested by curl http://localhost:8000.

Implementation in Go

package main

import (
    "fmt"
    "io"
    "log"
    "net/http"
)

func handler(w http.ResponseWriter, r *http.Request) {
    fmt.Fprintf(w, "Hello, World!")
}

func main() {
    http.HandleFunc("/", handler)
    go func() {
        err := http.ListenAndServe(":8000", nil)
        if err != nil {
            log.Fatalf("http.ListenAndServe failed: %v", err)
        }
    }()
    r, err := http.Get("http://localhost:8000/")
    if err != nil {
        log.Fatalf("http.Get failed: %v", err)
    }
    defer r.Body.Close()
    t, err := io.ReadAll(r.Body)
    if err != nil {
        log.Fatalf("io.ReadAll failed: %v", err)
    }
    fmt.Println(string(t))
}

Implementation in Go with graceful HTTP server shutdown

package main

import (
    "fmt"
    "io"
    "log/slog"
    "net/http"
    "os"
)

func handler(w http.ResponseWriter, r *http.Request) {
    fmt.Fprintf(w, "Hello, World!")
}

func configureLogging() {
    programLevel := new(slog.LevelVar)

    logger := slog.New(slog.NewTextHandler(os.Stdout, &slog.HandlerOptions{Level: programLevel}))
    slog.SetDefault(logger)

    if os.Getenv("DEBUG") != "" {
        programLevel.Set(slog.LevelDebug)
    }
}

func main() {
    configureLogging()

    http.HandleFunc("/", handler)

    addr := "localhost:8000"
    server := &http.Server{Addr: addr}

    done := make(chan struct{})
    go func(done chan<- struct{}) {
        slog.Debug("server.ListenAndServe", "addr", addr)
        err := server.ListenAndServe()
        if err != nil {
            slog.Debug("server.ListenAndServe", "error", err)
        }
        close(done)
    }(done)

    r, err := http.Get("http://" + addr)
    if err != nil {
        slog.Error("http.Get failed", "error", err)
        return
    }
    defer r.Body.Close()
    t, err := io.ReadAll(r.Body)
    if err != nil {
        slog.Error("io.ReadAll failed", "error", err)
        return
    }
    fmt.Println(string(t))
    err = server.Shutdown(nil)
    if err != nil {
        slog.Error("server.Shutdown failed", "error", err)
        return
    }
    <-done
}

Implementation in Typescript

Bun

import process from "node:process";

Bun.serve({ fetch: () => new Response("Hello World!"), port: 8000 });

console.log(await (await fetch("http://localhost:8000/")).text());
process.exit();

Deno

Deno.serve(
  { port: 8000, onListen: () => {} }, 
  () => new Response("Hello World!")
);
console.log(await(await fetch("http://localhost:8000/")).text());
Deno.exit();

The challenge for you is how you would implement this.

AWS Lambda in Deno or Bun

Alexander Demin — Thu, 04 Jul 2024 16:22:41 +0000

The article describes creating the AWS Lambda using purely Deno or Bun Javascript runtimes with zero external dependencies. We will use Deno by default, but the switch to Bun can be made via the RUNTIME variable (see Makefile).

Usually, to create an AWS lambda in Typescript, the code must be compiled in Javascript because AWS Lambda does not natively support Deno and Bun, only Node.

Some projects offer the flexibility of using Typescript directly in AWS Lambda, such as Deno Lambda.

However, we will implement our own custom AWS Lambda runtime to run Typescript by Deno or Bun and use AWS Lambda API directly.

The project comprises a Makefile for automation and clarity, Dockerfiles for containerization, and the lambda.ts file for the AWS Lambda function. That's all you need.

We will be building Docker images-based AWS Lambda deployment.

We start by explaining how to prepare AWS resources (image repository, role and policies, and lambda deployment).

We will use AWS CLI.

Part 1

You can skip this part of the article entirely if you are comfortable creating an AWS Elastic Container Repository named $(FUNCTION_NAME) (refer to Makefile) to prepare the lambda function named $(FUNCTION_NAME) to be created from the image, which we will build later.

You need to have .env file:

AWS_PROFILE=<YOU AWS PROFILE NAME>
AWS_ACCOUNT=<YOU AWS ACCOUNT>
AWS_REGION=<YOU AWS REGION>

The profile name allows AWS CLI to find your AWS_ACCESS_KEY_ID
and AWS_SECRET_ACCESS_KEY.

The .env file is included at the beginning of Makefile:

include .env
export

FUNCTION_NAME=lambda-ts-container
REPO = $(AWS_ACCOUNT).dkr.ecr.$(AWS_REGION).amazonaws.com

Once again, instead of using the Makefile file below, you can create the repository and manually prepare the AWS Lambda creation via AWS Console.

Create the depository:

create-repo:
  aws ecr create-repository \
  --profile $(AWS_PROFILE) \
  --repository-name $(FUNCTION_NAME)

make create-repo

ecr-login:
  aws ecr get-login-password --region $(AWS_REGION) \
  --profile $(AWS_PROFILE) \
  | docker login --username AWS --password-stdin $(REPO)

make ecr-login

Build, tag and push the image:

build-tag-push: build tag-push

build:
  docker build -t $(FUNCTION_NAME) \
  --platform linux/amd64 \
  -f Dockerfile-$(RUNTIME) .

tag-push:
  docker tag $(FUNCTION_NAME):latest \
  $(REPO)/$(FUNCTION_NAME):latest \
  docker push $(REPO)/$(FUNCTION_NAME):latest

make build-tag-push

Before creating the AWS Lambda, we need to create a role:

create-lambda-role:
  aws iam create-role \
  --profile $(AWS_PROFILE) \
  --role-name $(FUNCTION_NAME)-role \
  --assume-role-policy-document \
  '{"Version": "2012-10-17","Statement": [{ "Effect": "Allow", "Principal": {"Service": "lambda.amazonaws.com"}, "Action": "sts:AssumeRole"}]}'
  aws iam attach-role-policy \
  --profile $(AWS_PROFILE)
  --role-name $(FUNCTION_NAME)-role \
  --policy-arn arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole

make create-lambda-role

The role uses the AWSLambdaBasicExecutionRole policy, which allows the lambda to write logs for CloudWatch.

Finally, we create the lambda function:

create-lambda:
  aws lambda create-function \
  --function-name $(FUNCTION_NAME) \
  --role arn:aws:iam::$(AWS_ACCOUNT):role/$(FUNCTION_NAME)-role \
  --package-type Image \
  --code ImageUri=$(REPO)/$(FUNCTION_NAME):latest \
  --architectures x86_64 \
  --profile $(AWS_PROFILE) | cat

make create-lambda

We must create the AWS lambda URL and allow unauthenticated access to complete the lambda creation.

create-lambda-url:
  aws lambda create-function-url-config \
  --profile $(AWS_PROFILE) \
  --function-name $(FUNCTION_NAME) \
  --auth-type NONE 

create-lambda-invoke-permission:
  aws lambda add-permission \
  --profile $(AWS_PROFILE) \
  --function-name $(FUNCTION_NAME) \
  --action lambda:InvokeFunctionUrl \
  --statement-id FunctionURLAllowPublicAccess \
  --principal "*" \
  --function-url-auth-type NONE

make create-lambda-url create-lambda-invoke-permission

At this point, the AWS should be successfully created and deployed.

If you change the lambda source and want to deploy the update, you call:

deploy: build-tag-push update-image wait

update-image:
  SHA=$(shell make last-tag) && \
  echo "SHA=$(WHITE)$$SHA$(NC)" && \
  aws lambda update-function-code \
  --profile $(AWS_PROFILE) \
  --function-name $(FUNCTION_NAME) \
  --image $(REPO)/$(FUNCTION_NAME)@$$SHA \
  | jq -r '.CodeSha256'

status:
  @aws lambda get-function \
  --function-name $(FUNCTION_NAME) \
  --profile $(AWS_PROFILE) \
  | jq -r .Configuration.LastUpdateStatus

wait:
  @while [ "$$(make status)" != "Successful" ]; do \
    echo "wait a moment for AWS to update the function..."; \
    sleep 10; \
  done
  @echo "lambda function update complete"

make deploy

This command builds, tags and deploys a new image.

Let's invoke the function:

lambda-url:
  @aws lambda get-function-url-config \
  --function-name $(FUNCTION_NAME) \
  | jq -r '.FunctionUrl | rtrimstr("/")'

get:
  @HOST=$(shell make lambda-url) && \
  http GET "$$HOST/call?a=1"

make get

This command calls the lambda function via its public URL. The URL path is /call but can be anything with some query parameters. The path and query parameters will be provided to the function code and other standard HTTP-related information.

Other examples in Makefile invoke the function in different ways. For example, put- calls the data function in the request body.

put-json:
  @HOST=$(shell make lambda-url) && \
  http -b PUT "$$HOST/call?q=1" a=1 b="message"

put-text:
  @HOST=$(shell make lambda-url) && \
  http -b PUT "$$HOST/call?q=1" --raw='plain data'

get-418:
  @HOST=$(shell make lambda-url) && \
  http GET "$$HOST/call?a=1&status=418"

The code uses the http command from httpie.

Part 2

Let's look at the most exciting part -- the function's source code.

As I promised, we do not use any libraries. Instead, we use AWS Lambda API directly.

The AWS lambda lifecycle is a simple loop. The code below fetches the next function invocation event from the AWS API, passes it to the handler, and then sends the response to the AWS Lambda API response endpoint. That is it!

import process from "node:process";

const env = process.env;

const AWS_LAMBDA_RUNTIME_API = env.AWS_LAMBDA_RUNTIME_API || "?";
console.log("AWS_LAMBDA_RUNTIME_API", AWS_LAMBDA_RUNTIME_API);

const API = `http://${AWS_LAMBDA_RUNTIME_API}/2018-06-01/runtime/invocation`;

while (true) {
    const event = await fetch(API + "/next");

    const REQUEST_ID = event.headers.get("Lambda-Runtime-Aws-Request-Id");
    console.log("REQUEST_ID", REQUEST_ID);

    const response = await handler(await event.json());

    await fetch(API + `/${REQUEST_ID}/response`, {
        method: "POST",
        body: JSON.stringify(response),
    });
}

// This is a simplified version of the AWS Lambda runtime API.
// The full specification can be found at:
// https://docs.aws.amazon.com/lambda/latest/dg/services-apigateway.html

type APIGatewayProxyEvent = {
    queryStringParameters?: Record<string, string>;
    requestContext: { http: { method: string; path: string } };
    body?: string;
};

async function handler(event: APIGatewayProxyEvent) {
    const { method, path } = event.requestContext.http;

    const echo = {
        method,
        path,
        status: "200",
        queryStringParameters: {},
        runtime: runtime(),
        env: {
            ...env,
            AWS_SESSION_TOKEN: "REDACTED",
            AWS_SECRET_ACCESS_KEY: "REDACTED",
        },
        format: "",
        body: "",
    };

    if (event.queryStringParameters) {
        echo.queryStringParameters = event.queryStringParameters;
        echo.status = event.queryStringParameters.status || "200";
    }

    if (event.body) {
        try {
            echo.body = JSON.parse(event.body);
            echo.format = "json";
        } catch {
            echo.body = event.body;
            echo.format = "text";
        }
    }

    return {
        statusCode: echo.status,
        headers: { "Content-Type": "application/json" },
        body: JSON.stringify(echo),
    };
}

function runtime() {
    return typeof Deno !== "undefined"
        ? "deno " + Deno.version.deno
        : typeof Bun !== "undefined"
        ? "bun " + Bun.version
        : "maybe node";
}

For demonstration purposes, the handler returns the input data as the response.

NOTE: There are essential moments in the Dockerfile where we must configure the location for temporary files. The AWS Lambda container execution environment file system is read-only, and only the /tmp directory can be used for writing.

Let's discuss Dockerfiles to build the image.

FROM denoland/deno as deno

FROM public.ecr.aws/lambda/provided:al2

COPY --from=deno /usr/bin/deno /usr/bin/deno

# We need to set the DENO_DIR to /tmp because the AWS lambda filesystem 
# is read-only except for /tmp. Deno may need to write to its cache.
ENV DENO_DIR=/tmp

COPY lambda.ts /var/task/

ENTRYPOINT [ "/usr/bin/deno" ]
CMD [ "run", "-A", "--no-lock", "/var/task/lambda.ts"]

Dockerfile uses the official AWS base image public.ecr.aws/lambda/provided:al2.

This image comes with the AWS Lambda Runtime Client preinstalled. This client runs in the background and proxies the requests from the lambda function loop to AWS endpoints. The AWS_LAMBDA_RUNTIME_API variable points to localhost with a port on which the Lambda Runtime Client listens.

This concludes the article.

By default, Makefile uses Deno (RUNTIME=deno). The RUNTIME variable can be set to bun as a drop-in change, so no other changes are required.

For convenience, the handler reports what runtime it is running on in the runtime field.

Resources

The links to the sources of the files from this article:

Serve file system browser as a web page

Alexander Demin — Thu, 28 Dec 2023 08:33:55 +0000

This demo application exposes the /fs/ endpoint serving the index of the file system.

The application uses Deno runtime, Hono web framework and nunjucks templates.

Usage

deno task dev

Implementation

The crucial part of the application. The complete source is in the repo.

app.get(FS + "/*", (c) => {
    const path_ = decodeURI(c.req.path);
    const path = resolve(join("/", path_.slice(FS.length), "/"));
    const stat = Deno.statSync(path);
    if (stat.isDirectory) return c.html(directory(path));
    else {
        const { content, type } = file(path);
        return new Response(content, { headers: { "content-type": type } });
    }
});

The example of the page

Async file download in JavaScript and Python

Alexander Demin — Fri, 22 Dec 2023 15:59:02 +0000

Async file download in JavaScript and Python

https://github.com/begoon/tmpz/tree/main/async-download

Goals

Implement async file download in JavaScript and Python.
Compare the download speed.
Compare the order of downloads.
Compare the error handling.

Python

poetry install
poetry shell

python main.py

Python Results

downloading http://ipv4.download.thinkbroadband.com/10MB.zip
downloading http://ipv4.download.thinkbroadband.com/10MB.zip
downloading http://ipv4.download.thinkbroadband.com/10MB.zip
downloading http://ipv4.download.thinkbroadband.com/10MB.zip
downloading http://ipv4.download.thinkbroadband.com/10MB.zip
downloading http://ipv4.download.thinkbroadband.com/10MB.zip
downloading http://ipv4.download.thinkbroadband.com/10MB.zip
downloading http://ipv4.download.thinkbroadband.com/10MB.zip
downloading http://ipv4.download.thinkbroadband.com/10MB.zip
downloading http://ipv4.download.thinkbroadband.com/10MB.zip
2: downloaded http://ipv4.download.thinkbroadband.com/10MB.zip, 10485760 bytes
1: downloaded http://ipv4.download.thinkbroadband.com/10MB.zip, 10485760 bytes
5: downloaded http://ipv4.download.thinkbroadband.com/10MB.zip, 10485760 bytes
3: downloaded http://ipv4.download.thinkbroadband.com/10MB.zip, 10485760 bytes
0: downloaded http://ipv4.download.thinkbroadband.com/10MB.zip, 10485760 bytes
9: downloaded http://ipv4.download.thinkbroadband.com/10MB.zip, 10485760 bytes
8: downloaded http://ipv4.download.thinkbroadband.com/10MB.zip, 10485760 bytes
7: downloaded http://ipv4.download.thinkbroadband.com/10MB.zip, 10485760 bytes
6: downloaded http://ipv4.download.thinkbroadband.com/10MB.zip, 10485760 bytes
[[10485760, 0], [10485760, 1], [10485760, 2], [10485760, 3], Exception('random exception: n=4'), [10485760, 5], [10485760, 6], [10485760, 7], [10485760, 8], [10485760, 9]]
10 files downloaded in 23.77 seconds
throughput 3.79 MB/s

JavaScript

deno run -A main.js

bun main.js

JavaScript results

downloading http://ipv4.download.thinkbroadband.com/10MB.zip
downloading http://ipv4.download.thinkbroadband.com/10MB.zip
downloading http://ipv4.download.thinkbroadband.com/10MB.zip
downloading http://ipv4.download.thinkbroadband.com/10MB.zip
downloading http://ipv4.download.thinkbroadband.com/10MB.zip
downloading http://ipv4.download.thinkbroadband.com/10MB.zip
downloading http://ipv4.download.thinkbroadband.com/10MB.zip
downloading http://ipv4.download.thinkbroadband.com/10MB.zip
downloading http://ipv4.download.thinkbroadband.com/10MB.zip
downloading http://ipv4.download.thinkbroadband.com/10MB.zip
0: downloaded http://ipv4.download.thinkbroadband.com/10MB.zip, 10485760 bytes
8: downloaded http://ipv4.download.thinkbroadband.com/10MB.zip, 10485760 bytes
7: downloaded http://ipv4.download.thinkbroadband.com/10MB.zip, 10485760 bytes
3: downloaded http://ipv4.download.thinkbroadband.com/10MB.zip, 10485760 bytes
5: downloaded http://ipv4.download.thinkbroadband.com/10MB.zip, 10485760 bytes
1: downloaded http://ipv4.download.thinkbroadband.com/10MB.zip, 10485760 bytes
6: downloaded http://ipv4.download.thinkbroadband.com/10MB.zip, 10485760 bytes
[
  { status: "fulfilled", value: { n: 0, sz: 10485760 } },
  { status: "fulfilled", value: { n: 1, sz: 10485760 } },
  {
    status: "rejected",
    reason: Error: random error: n=2
    at download_file (file:///Users/...[reducted].../async-download/main.ts:9:36)
    at file:///Users/...[reducted].../async-download/main.ts:21:54
    at Array.map (<anonymous>)
    at main (file:///Users/...[reducted].../async-download/main.ts:21:40)
    at file:///Users/...[reducted].../async-download/main.ts:36:1
  },
  { status: "fulfilled", value: { n: 3, sz: 10485760 } },
  {
    status: "rejected",
    reason: Error: random error: n=4
    at download_file (file:///Users/...[reducted].../async-download/main.ts:9:36)
    at file:///Users/...[reducted].../async-download/main.ts:21:54
    at Array.map (<anonymous>)
    at main (file:///Users/...[reducted].../async-download/main.ts:21:40)
    at file:///Users/...[reducted].../async-download/main.ts:36:1
  },
  { status: "fulfilled", value: { n: 5, sz: 10485760 } },
  { status: "fulfilled", value: { n: 6, sz: 10485760 } },
  { status: "fulfilled", value: { n: 7, sz: 10485760 } },
  { status: "fulfilled", value: { n: 8, sz: 10485760 } },
  {
    status: "rejected",
    reason: Error: random error: n=9
    at download_file (file:///Users/...[reducted].../async-download/main.ts:9:36)
    at file:///Users/...[reducted].../async-download/main.ts:21:54
    at Array.map (<anonymous>)
    at main (file:///Users/...[reducted].../async-download/main.ts:21:40)
    at file:///Users/...[reducted].../async-download/main.ts:36:1
  }
]
10 files downloaded in 19 seconds
throughput: 3.78 MB/s

Conclusion

Both Python and JavaScript download speeds are about the same.
As a result, the order of resolved Promises in JavaScript and Async Tasks in Python is preserved.

Compile C to WebAssembly (WASM) and run it in the browser

Alexander Demin — Sun, 30 Jul 2023 20:48:33 +0000

Introduction

A popular toolchain to compile C to WASM is emscripten. It provides bindings to the standard C library and other interfaces, like SDL2, and allows to compile C code to WASM and run it in the browser.

Unfortunately, emscripten is complex. It generates a lot of JavaScript code boilerplate in HTML and JavaScript files.

If you only need to compile the C code, which does not use the standard C library or other interfaces, you can use a simpler toolchain -- clang with llc.

The example below demonstrates how to compile C code to WASM, run it in the browser and communicate with JavaScript using standard C types including pointers.

Prerequisites

On Mac, it only requires to install llvm from brew to enable the clang compiler to generate WASM code:

brew install llvm

Then the path to llc toolchain must be added to the PATH environment:

export PATH=/opt/homebrew/opt/llvm/bin:$PATH

After that, the clang compiler should support the --target=wasm32 target:

llc --version | grep wasm

Compile C to WASM

main.c:

unsigned char mem[0x10000];

const char *const upper(char *const str, const int sz)
{
    for (int i = 0; i < sz; i++)
        if (str[i] >= 'a' && str[i] <= 'z')
            str[i] -= 0x20;
    return str;
}

compile it to WASM:

clang \
--target=wasm32 \
--no-standard-libraries \
-Wl,--export-all -Wl,--no-entry \
-o main.wasm \
main.c

The command creates a file main.wasm which is a binary WASM module.

This trivial example demonstrates basic needs to the simple C-to-WASM interface:

an ability to use pre-allocated static memory buffer (for example, of size 64KB)
an ability to call C functions from JavaScript and pass primitive types (integers, pointers, etc.) as arguments
an ability to return primitive types from C functions
an ability to observe the memory buffer changes from JavaScript

The standard C library is not linked to the WASM module, so the C functions should not use any standard C functions.

Interestingly, the result of the function upper(), which is a pointer, is returned to JavaScript as a number, not as a pointer. This number is an offset from the beginning of the WebAssembly memory buffer on the JavaScript side.

C deals with pointers, but JavaScript sees the memory buffer as a flat array and uses offsets to access its elements.

In fact, WASM "direct memory access" is not what it literally means. WASM memory is a regular JavaScript object, which is a flat array of bytes. Its lifecycle is managed by JavaScript garbage collector as any other JavaScript objects.

Run WASM in the browser

Use the following main.html file to run the WASM module in the browser:

<!DOCTYPE html>

<html>
    <body>
        <script type="module">
            const log = console.log;
            console.log = (...x) => {
                document.body.innerHTML += x.join(" ") + "<br>";
                log(...x);
            };
            const wa = await WebAssembly.instantiateStreaming(
                fetch("main.wasm"),
                { js: { mem: new WebAssembly.Memory({ initial: 0 }) } }
            );
            const memPtr = wa.instance.exports.mem.value;
            console.log("memPtr", memPtr);

            const mem = new Uint8Array(wa.instance.exports.memory.buffer);

            const str = "Hello, world!";
            new TextEncoder().encodeInto(
                str,
                mem.subarray(memPtr, memPtr + str.length)
            );
            console.log(
                "upper()",
                wa.instance.exports.upper(memPtr, str.length)
            );
            console.log(
                new TextDecoder().decode(
                    mem.subarray(memPtr, memPtr + str.length)
                )
            );
        </script>
    </body>
</html>

The main.html and the main.wasm files need to be served by a web server.

For instance, by by Python's SimpleHTTPServer:

python -m SimpleHTTPServer

or VSCode's Live Server extension.

You should see something like this in the browser and in the devtool console:

memPtr 1024
upper() 1024
HELLO, WORLD!

1024 is in fact "a pointer". 1024 is an offset from the beginning of the mem C array in the WASM module memory.

The upper() function converts "Hello, world!" to upper case and returns the same "pointer" 1024 to JavaScript.

That is it. The WASM module is compiled and run in the browser.

Writing a Telegram bot with SvelteKit and Vercel

Alexander Demin — Sat, 08 Jul 2023 14:29:11 +0000

Let's write a Telegram bot with SvelteKit. The bot will run in Vercel, which provides zero-configuration support for SvelteKit.

Assume we already have a Telegram bot registered. Its token is in the BOT_TOKEN environment variable.

Our bot will use a webhook type of deployment, not polling.

The heart of the Telegram bot operating with the webhook is the POST endpoint, receiving callbacks from the Telegram servers. After the POST endpoint receives the request and deserializes the payload from JSON, it passes it to the "node-telegram-bot-api" library to handle.

Our bot will be trivial only to demonstrate how to handle the requests from Telegram in SvelteKit. It will echo back the received text messages.

After creating a project with "npm create svelte", install the Telegram library with "node-telegram-bot-api" and add the following configuration to "svelte.config.js":

const config = {
        ...
        kit: {
            ...
            csrf: {
                checkOrigin: false,
            }
       }
};

let's create a "+server.ts" file in the "./routes" directory.

import { bot } from "./bot";

export async function POST({ request }) {
    try {
        bot.processUpdate(await request.json());
        return new Response("", { status: 200 });
    } catch (e) {
        console.error(e);
        return new Response("", { status: 500 });
    }
}

The "bot.ts" file implements the echoing of incoming messages.

export const bot = new TelegramBot(BOT_TOKEN);
console.log("bot", BOT_TOKEN);

export const history: any[] = [];
bot.on("message", async (msg) => {
    try {
        await bot.sendMessage(msg.chat.id, msg.text + ", and?");
        history.push({ when: new Date(), msg });
    } catch (e) {
        console.error(e);
    }
});

That's it.

Once the code is pushed to a git repo and deployed by Vercel, for example, on "https://sveltekit-bot.vercel.app", we need to set a webhook allowing Telegram to deliver the updates to the bot.

It needs to be done just once:

curl -X POST https://api.telegram.org/bot$(BOT_TOKEN)/setWebhook \
    -H "Content-type: application/json" \
    -d '{"url": "https://sveltekit-bot.vercel.app"}'

After that, we can start chatting with the bot!

NOTE: It is important to understand that this bot deployment to Vercel is stateless. When there is no traffic, Vercel stops the endpoint and kicks it off again when the traffic is coming. There is no data persistence
in the bot with this implementation. In a real bot, we may use external storage, like a database, to persist the bot's state.

The full source of the project is available at
https://github.com/begoon/sveltekit-bot/.

In the project, there is a trivial in-memory storage for messages and a page to print them for demonstration purposes. After the inactivity period, the deployment restarts by Vercel, and the message storage resets.

How to glue different applications inside a docker container or implement a reverse proxy in Go

Alexander Demin — Mon, 08 May 2023 19:49:12 +0000

Multiple web applications in the docker container may be implemented in different languages or frameworks. They all may need to be served from the container on the same external port so that it looks like a single application from outside the container.

You may have a third-party application to run in the container, which you cannot change, and you have to run it as is. At the same time, you would like to have your custom endpoints alongside that application.

For example, the main application runs from the root / and uses all required URLs below the root. You may want to have one URL, for example, /health, which you want to serve yourself.

The solution for this configuration is the reverse proxy, which routes requests inside the container to different ports, and even redirects the requests to external URLs.

Enter the reverse proxy

In the article, we will "glue" three web applications implemented in Python, Node and Go. All three will be deployed in the same container, and each application will be invoked depending on the URL.

The default application is in Python. This application serves from the root (/) and all other URLs not handed by other applications. This application runs on port 9000 inside the container. This port is not exposed outside the container.

The application in Node runs on port 9100 and receives the requests with the /node prefix. This port also is not exposed outside the container.

The application in Go runs on port 8000 and receives requests with the /go prefix. This port is exposed as the container port.

Additionally, one more special prefix, /google, will redirect to Google Search.

Recap:

/node* goes to the Node application
/go* goes to the Go application
/google goes to "https://google.com".
/* - everything else goes to the default Python application

The application in Go also implements the reverse proxy mechanism, the essence of the example.

Note:

The code in this article is for the demonstration purpose only. The configuration may be hard coded. In the actual production deployment, there should be a better abstraction on configuring Dockerfile and the applications.

The code of the applications is also the bare minimum. Each endpoint will print some text and the received URL.

Python application

from fastapi import FastAPI, Request
from fastapi.responses import PlainTextResponse

app = FastAPI()

@app.get("/{path:path}", response_class=PlainTextResponse)
async def root(request: Request, path: str):
    return f"I'm Python!\r\n[{path}]\r\n"

Node application

const { createServer } = require("http");

createServer((req, res) => {
    res.setHeader("content-type", "text/plain");
    res.end(`I'm Node!\r\n[${req.url}]\r\n`);
}).listen(process.env.PORT || 9000);

Go application with a built-in proxy

This is the meat and potatoes of the article.

It uses the "ReverseProxy" object and the "NewSingleHostReverseProxy" utility from the standard Go library. The application has no third-party external dependencies.

The code below starts the HTTP listener on port 8000, which will be exposed outside the container, and redirects the traffic to other ports according to the logic above.

package main

import (
    "fmt"
    "log"
    "net/http"
    "net/http/httputil"
    "net/url"
    "os"
    "strings"
)

func main() {
    defaultURL, err := url.Parse("http://localhost:9000")
    if err != nil {
        log.Fatal(fmt.Errorf("error parsing default URL: %v", err))
    }

    nodeURL, err := url.Parse("http://localhost:9100")
    if err != nil {
        log.Fatal(fmt.Errorf("error parsing node URL: %v", err))
    }

    googleURL, err := url.Parse("https://google.com")
    if err != nil {
        log.Fatal(fmt.Errorf("error parsing google's URL: %v", err))
    }

    http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
        if q, found := strings.CutPrefix(r.URL.Path, "/google"); found {
            proxy := &httputil.ReverseProxy{
                Rewrite: func(r *httputil.ProxyRequest) {
                    r.SetURL(googleURL)
                    r.Out.URL.Path = "/" + q
                },
            }
            proxy.ServeHTTP(w, r)
            return
        }
        if strings.HasPrefix(r.URL.Path, "/go") {
            w.Write([]byte(fmt.Sprintf("I'm Go!\r\n[%v]\n", r.URL.Path)))
            return
        }
        if strings.HasPrefix(r.URL.Path, "/node") {
            httputil.NewSingleHostReverseProxy(nodeURL).ServeHTTP(w, r)
            return
        }
        httputil.NewSingleHostReverseProxy(defaultURL).ServeHTTP(w, r)
    })
    port := os.Getenv("PORT")
    if port == "" {
        port = "8000"
    }
    if err := http.ListenAndServe(":"+port, nil); err != nil {
        log.Fatal(err)
    }
}

Container startup script

This script starts all three applications on different ports and provides the required environment variables. This script is the container entry point.

#!/usr/bin/env bash

(source .venv/bin/activate && uvicorn main:app --port 9000) &
(PORT=9100 ./node ./main.js) &
(PORT=8000 ./proxy) &

wait
exit $?

Dockerfile

Dockerfile is multistaged because it needs to collect artefacts from the different applications in one container.

ARG GO_VERSION=1.20

FROM golang:${GO_VERSION}-alpine AS build-proxy
WORKDIR /app

COPY main.go .
RUN go build -o proxy main.go

# ---
FROM python:3-slim AS build-python

WORKDIR /app

COPY main.py .
RUN python -m venv .venv
RUN . .venv/bin/activate && pip install uvicorn fastapi

# ---
FROM node:20-bullseye-slim AS build-node

WORKDIR /app

COPY main.js .
RUN cp `which node` .

# ---
FROM python:3-slim
WORKDIR /app

COPY --from=build-python /app .
COPY --from=build-node /app .
COPY --from=build-proxy /app/proxy .
COPY ./run.sh .

CMD ["./run.sh"]

Build the docker image

The image's name is "zoo".

docker build -t zoo .

Running the container

This command runs the container in interactive mode and with --rm to be deleted when it stops.

docker run -p 8000:8000 --rm -it zoo

Testing

The container is up the running. It listens on port 8000, so we can test it.

Hitting the Python application:

curl http://localhost:8000

I'm Python!
[]

curl http://localhost:8000/a/b/c

I'm Python!
[a/b/c]

Hitting the Node application:

curl http://localhost:8000/node

I'm Node!
[/node]

curl http://localhost:8000/node/a/b/c

I'm Node!
[/node/a/b/c]

Hitting the Go application:

curl http://localhost:8000/go

I'm Go!
[/go]

curl http://localhost:8000/go/a/b/c

I'm Go!
[/go/a/b/c]

Finally, you can put the http://localhost:8000/google/?q=abc URL into the browser, which will forward you to Google Search.

This is it! We have implemented the configurable reverse proxy in Go.

It glues together three applications written in different languages, and they run inside one container as a single deployable unit.

Makefile

For convenience, a Makefile that can build the image and run the container:

make

Run tests above:

make test

Performance

Extra data transfer may be necessary when the traffic goes through the proxy. We say "maybe" because, in reality, the proxy can propagate actual data copying between sockets to the kernel. The approach is called Zero-Copy networking. It eliminates the overhead because all network listeners run on the same machine and are controlled by the same kernel.

The Go standard library on Linux does precisely that.

Extra copying is needed when the proxy redirects to the external URL, but this is not the intended use case for this application.

Links

The sources are available at https://github.com/begoon/go-reverse-proxy.

Simple encrypted storage for python applications

Alexander Demin — Fri, 23 Sep 2022 10:20:47 +0000

This article is about a simple encrypted storage for python applications using AES/GCM and base64.

Introduction

Sometimes, you have secret constants in the code (passwords, tokens, API keys etc.).

The standard approach is to set them via environment variables. This approach is safe, but it comes with the inconvenience of managing your setup. You must ensure that all required environment variables are present before the code runs.

In CI/CD, you would use a secrets storage or manager. In deployment, we need to figure out how to inject the environment variables, whether we work on staging, production etc.

It may be more convenient if the code itself can manage everything. It needs to know the target environment (CI/CD, staging, production etc.) and only one key/password to unlock all other secret settings.

The approach below will allow shipping all your settings (including secret or sensitive ones) within the code because we will encrypt them at rest.

When the code runs, it decrypts the secrets in memory using the key, which it takes from the environment as the external setting.

To fully configure your code, you will only need two environment variables to set beforehand:

ENVIRONMENT (for example, dev, or ci or prod)
ENVIRONMENT_PASSWORD

Setup

To configure the secrets vault, you need to set one environment variable -- ENVIRONMENT_PASSWORD.

This variable is used to encrypt and decrypt your secrets.

You always keep the value of this variable secret. You set it up locally in your development environment or inject it into your CI/CD via a secrets manager/storage provided by your hosting (for example, GitHub Secrets).

Prepare your secrets

We put the secrets in the code in encrypted form. To encrypt the secrets, you can use the following CLI:

python vault_cli.py --help

For example:

python vault_cli.py encrypt my_secret_password

Output:

aes:4XgUg6US2yzMUf6367PNGlQb0RsVXV5I20VqJhwGTVOcgcdLQdj3WLOnISzC82xaPmE=

You use this value in the code.

The PASSWORD and KEYvalues in the example below were encrypted this way.

Example

Before running this example, you need to set ENVIRONMENT_PASSWORD:

export ENVIRONMENT_PASSWORD=super_secret

The code below contains both plain and encrypted values in settings.
The setting() function returns a given value by name.

At the start, the code calls vault.check() from vault.py to verify the consistency of the vault. If the value of ENVIRONMENT_PASSWORD is incorrect, the function will throw an exception.

The vault.check() function uses a hardcoded value. It is not used in the user code. It is the self-check purpose only.

Are you curious what that magic DIGEST variable contains? :). You can decrypt it via python vault_cli.py decrypt command.

import vault

settings = {
    "USERNAME": "staff",
    "PASSWORD": "aes:ANyPgY1HTuJPS2mo2Xze/+MUZl8992deZdFHYml67QKWCJ0NzX4=",
    "KEY": "aes:cGJKLBYfGXMLX/oTulGxZFz97ap+SZ3uJBODogfixTzjnIW82g==",
}


def setting(name: str) -> str:
    return vault.decrypt(settings[name])


def main():
    print('login credentials')
    print('username =', setting('USERNAME'))
    print('password =', setting('PASSWORD'))
    print('key =', setting('KEY'))


if __name__ == '__main__':
    vault.check()
    main()

Why is it all useful?

It is safe to commit this code to public repositories. All sensitive values are protected by AES/GCM encryption.

Of course, you should NEVER expose ENVIRONMENT_PASSWORD.

Security considerations

We need to understand that this approach may have downsides.

If your environment password leaks, all environments using it are compromised. However, the threat of leaking the environment password is not much different from the threat when your set of environment settings leaks.

To mitigate the exposure of all environments to be compromised, you can use different environment passwords for different environments.

It will require minor code changes, but I leave it to you as a home exercise.

Finally, you should know how encryption works in this method.

We use AES with 256 bit key in GCM mode. GCM does not require padding. It also allows checking the consistency of the data on decryption.

The 256-bit key is generated via SHA256 from the password.

nonce (initial vector for AES) is generated randomly. It guarantees
that same data will look different when encrypted plays the role of
a salt.

AES/GCM returns the cypher text (encrypted data), nonce and tag. All three participate in decryption.

The encrypt function concatenates all three values, wraps the result
in base64 and prepends it with aes: prefix.

The prefix allows to recognise encrypted values and skip the encryption
for non-encrypted ones.

def get_aes_key(password: str) -> bytes:
    return hashlib.sha256(password.encode()).digest()

def encrypt_value_aes(value: bytes, password: str) -> bytes:
    nonce = secrets.token_bytes(16)
    aes = AES.new(get_aes_key(password), AES.MODE_GCM, nonce=nonce)
    encrypted, tag = aes.encrypt_and_digest(value)
    return nonce + tag + encrypted

def encrypt(value: str, password: Optional[str] = None) -> str:
    encryped = encrypt_value_aes(value.encode(), get_password(password))
    return 'aes:' + base64.b64encode(encryped).decode()

The decryption works in reverse.

Links

All the code from this article is in github.com/begoon/pyvault GitHub repository.

Writing Telegram bots from command line

Alexander Demin — Tue, 13 Sep 2022 15:49:05 +0000

Telegram is a fantastic messaging service. It suits social communications well, but it can also be used nicely in different automation scenarios, for example, in DevOps.

Enter Telegram bots. These are special Telegram accounts, which are usually used programmatically. Software controls the behaviour of such accounts, allowing them to communicate with other Telegram accounts (individuals, groups or channels).

The article below will show how to start using Telegram bots programmatically. To begin with, you need only one tool from your command line environment. It is curl.

Other tools are available for similar purposes, for example, HTTPie. Technically, you can match curl syntax to HTTPie almost one-to-one. Still, I will be using curl, because it is standard and available by default in the majority and UNIX-based environments and macOS. On Windows, you may need to install it separately. It would be best if you also were careful with double quotes and the syntax of the environment variables substitution in the commands below because cmd or PowerShell interpret them differently.

First, you must create a Telegram if you don't have one. It takes less than a minute. Go to your Telegram client on any platform, find the @BotFather and open the chat with it. @BotFather is the bot provided by Telegram, and it is used to create and manage all other bots.

Type /newbots, follow simple steps and create a bot.

In the example above, I have created a new bot called my_supercool_shiny_bot. By the Telegram rules, the bot names must always end with bot. Pick the name accordingly.

Telegram generated the unique token of this bot 5387301065:AAEWZqC49ilJ9Xds93sP7gU_70OdAfIiTSA.

IMPORTANT NOTE: Please, keep the token of your bot secret. Never commit it to public repos, and always pass it as the parameter to your code, for example, via an environment variable.

If someone knows your bot's token, they can communicate with Telegram on behalf of your bot, and you will have to revoke the token and regenerate it.

Don't worry. It is impossible to hack into your main Telegram account via a leaked bot token, but the bot itself will be compromised.

I am using the bot token openly in this article for demonstration purposes, but by the time the article will be published, the bot will be removed, and the token will no longer be valid.

Ok, let's set an environment variable with the bot token first:



export BOT_TOKEN="5387301065:AAEWZqC49ilJ9Xds93sP7gU_70OdAfIiTSA"

Let's start talking to the bot by requesting its details via the getMe command:



curl https://api.telegram.org/bot$BOT_TOKEN/getMe

It will respond with:



{
   "ok":true,
   "result":{
      "id":5387301065,
      "is_bot":true,
      "first_name":"my_supercool_shiny_bot",
      "username":"my_supercool_shiny_bot",
      "can_join_groups":true,
      "can_read_all_group_messages":false,
      "supports_inline_queries":false
   }
}

We see that the bot responds with details about itself. The main thing here is its Telegram id 5387301065. This number represents the bot within Telegram API. Also, we see the bot username - my_supercool_shiny_bot.

Now let's send a message from the bot to a Telegram user, for example, you. To do that, we need to know your Telegram id (it's not your Telegram username). We can find it out if you don't remember it.

In your Telegram GUI client find our new bot via search by its name.

In my case, it is my_supercool_shiny_bot. Initiate the chat with the bot by pressing the Start button.

Let's now send a message from you to the bot. Type, for example, "TEST" in the chat and send it.

Now, let's receive that message in the bot via the getUpdates command:



curl "https://api.telegram.org/bot$BOT_TOKEN/getUpdates"

The response would be:



{
   "ok":true,
   "result":[
      {
         "update_id":754613580,
         "message":{
            "message_id":1,
            "from":{
               "id":123456789,
               "is_bot":false,
               "first_name":"FIRSTNAME",
               "last_name":"LASTNAME",
               "username":"USERNAME",
               "language_code":"en"
            },
            "chat":{
               "id":123456789,
               "first_name":"FIRSTNAME",
               "last_name":"LASTNAME",
               "username":"vegoon",
               "type":"private"
            },
            "date":1663079584,
            "text":"/start",
            "entities":[
               {
                  "offset":0,
                  "length":6,
                  "type":"bot_command"
               }
            ]
         }
      },
      {
         "update_id":754613581,
         "message":{
            "message_id":2,
            "from":{
               "id":123456789,
               "is_bot":false,
               "first_name":"FIRSTNAME",
               "last_name":"LASTNAME",
               "username":"USERNAME",
               "language_code":"en"
            },
            "chat":{
               "id":123456789,
               "first_name":"FIRSTNAME",
               "last_name":"LASTNAME",
               "username":"USERNAME",
               "type":"private"
            },
            "date":1663079706,
            "text":"TEST"
         }
      }
   ]
}

There two updates from Telegram to the bot. The first message is /start. It happened when we initiated the chat with the bot. The second message with "text":"TEST" is the text we sent in the GUI client.

Now the bot sees your message!

Also, there is another essential value in this response - your Telegram user id. It is the id field in JSON. In this example - 123456789. Let's remember it.

Let's send the response back from the bot to you via the sendMessage command (replace 123456789 with your Telegram user id):



curl -X POST \
-H 'Content-Type: application/json' \
-d '{"chat_id": "123456789", "text": "Reply from the bot to your TEST sent via curl"}' \
https://api.telegram.org/bot$BOT_TOKEN/sendMessage

The response would be:



{
   "ok":true,
   "result":{
      "message_id":3,
      "from":{
         "id":5387301065,
         "is_bot":true,
         "first_name":"my_supercool_shiny_bot",
         "username":"my_supercool_shiny_bot"
      },
      "chat":{
         "id":123456789,
         "first_name":"FIRSTNAME",
         "last_name":"LASTNAME",
         "username":"USERNAME",
         "type":"private"
      },
      "date":1663080612,
      "text":"Reply from the bot to your TEST sent via curl"
   }
}

In the Telegram GUI client, you will see the response:

The bot has responded to you!

Reflect on what we have just done for a second. Just in the few commands, we communicated between the bot and you. Now imagine, you can programmatically send updates from your DevOps pipelines or other running services to operators via Telegram! There are lots of different possible use cases.

That's it.

We only scratched the surface of Telegram bots. For further details, consider the official Telegram Bot API documentation.

But on the fundamental level, the techniques we looked at in this article can be used for all other functions from the Telegram Bot API.

Final notes.

In this article, we created a bot from scratch. You could also use an existing bot if you know its token.

In this case, when you issued the getUpdates command, you might receive an error like:



{
   "ok":false,
   "error_code":409,
   "description":"Conflict: can't use getUpdates method while webhook is active; use deleteWebhook to delete the webhook first"
}

It means that some software is already operating the bot via the webhook. You need to delete the webhook first to be able to retrieve the bot updates via the getUpdates command directly. I would recommend the check with the bot's owner first that the bot is not used in essential operations, and you can remove the webhook temporarily.

To remove the webhook, use the deleteWebhook command.



curl "https://api.telegram.org/bot$BOT_ID/deleteWebhook"

The response for the removal would be:



{
   "ok":true,
   "result":true,
   "description":"Webhook was deleted"
}

After that, you can use the getUpdates command directly.

GitHub as a database

Alexander Demin — Tue, 13 Sep 2022 09:00:36 +0000

git is indeed a database. GitHub is a remote database powered by git.

I needed a way to keep information about certain important events in my code nicely saved for later analysis. What can be better than committing them to a VCS? Timestamps, commit descriptions etc.

I used local git first and then switched to GitHub. GitHub provides API for all its functionality.

The little code below demonstrates how this approach works.

It needs two things to be set: GITHUB_TOKEN, which can be generated in your GitHub account and the repo variable with the repository name.

It upserts a new file to create a file. Then it upserts it again to modify it. Then it deletes the file.

The repository log nicely keeps all these actions in the commit history.

Note: The PyGithub package needs to be installed first:

pip instal pygithub

Code:

import os
from typing import Optional, Union

import github
from github.Repository import Repository


def get_repo(repo: str) -> Repository:
    assert repo, 'repository name is missing'
    g = github.Github(os.environ['GITHUB_TOKEN'])
    return g.get_repo(repo)


def upsert_file(
    name: str,
    body: str,
    message: Optional[str] = None,
    *,
    repo: Optional[Union[Repository, str]] = None,
    branch: Optional[str] = "main",
    verbose: Optional[bool] = False,
):
    r = repo if isinstance(repo, Repository) else get_repo(repo)
    try:
        description_ = message or f'Update {name}'
        current = r.get_contents(name, ref=branch)
        current = r.update_file(
            current.path,
            description_,
            body,
            current.sha,
            branch=branch,
        )
        if verbose:
            print(current)
    except github.GithubException:
        message = message or f'Create {name}'
        created = r.create_file(name, message, body, branch=branch)
        if verbose:
            print(created)


def delete_file(
    name: str,
    message: str = None,
    *,
    repo: Optional[Union[Repository, str]] = None,
    branch: str = "main",
    verbose: Optional[bool] = False,
):
    r = repo if isinstance(repo, Repository) else get_repo(repo)
    message = message or f'Delete {name}'
    current = r.get_contents(name, ref=branch)
    deleted = r.delete_file(
        current.path,
        message,
        current.sha,
        branch=branch,
    )
    if verbose:
        print(deleted)


assert os.getenv('GITHUB_TOKEN'), 'Set GITHUB_TOKEN variable'

repo = "<YOUR_GITHUB_NAME>/<REPO_NAME>"

upsert_file("README.md", "NEW BODY", repo=repo, verbose=True)
upsert_file("README.md", "UPDATED BODY", repo=repo, verbose=True)

delete_file("README.md", repo=repo, verbose=True)

Execute it by:

python main.py

It prints something like:

{'content': ContentFile(path="README.md"), 'commit': Commit(sha="a6c540fec9b1b02e21acbb0ddd790efb6b7cb33f")}
{'commit': Commit(sha="2436e7ff2692a9af398dabd9eb9d1eee0f821954"), 'content': ContentFile(path="README.md")}
{'commit': Commit(sha="31fefb51e3510071777e4f4c8a0971de0a184f78"), 'content': NotSet}

Go to your GitHub repository and check the commits.

DEV Community: Alexander Demin

Compiler for "Easy" language from "Etudes for Programmers" book (1978)

EASY language compiler

EASY language intro

Implementation

Compiler test pipeline

Testing and running examples

Prerequisites

Examples

Compiler internals overview

Outro

Related

"Hello World" in HTTP way

Problem statement

Implementation in Go

Implementation in Go with graceful HTTP server shutdown

Implementation in Typescript

Bun

Deno

The challenge for you is how you would implement this.

AWS Lambda in Deno or Bun

Part 1

Part 2

Resources

Serve file system browser as a web page

Usage

Implementation

The example of the page

Async file download in JavaScript and Python

Async file download in JavaScript and Python

Goals

Python

Python Results

JavaScript

JavaScript results

Conclusion

Compile C to WebAssembly (WASM) and run it in the browser

Introduction

Prerequisites

Compile C to WASM

Run WASM in the browser

Writing a Telegram bot with SvelteKit and Vercel

How to glue different applications inside a docker container or implement a reverse proxy in Go

Enter the reverse proxy

Python application

Node application

Go application with a built-in proxy

Container startup script

Dockerfile

Build the docker image

Running the container

Testing

Makefile

Performance

Links

Simple encrypted storage for python applications

Introduction

Setup

Prepare your secrets

Example

Security considerations

Links

Writing Telegram bots from command line

GitHub as a database