DEV Community: Ole Petersen

Best way to build customizable result list filter

Ole Petersen — Fri, 22 Apr 2022 13:48:55 +0000

Given a long list of some type of data (for example products, images, employees, articles...), what's your favorite way to filter and sort the items in a user interface?

How do you manage notifications in your apps?

Ole Petersen — Thu, 14 Apr 2022 08:49:47 +0000

When you're working on a fullstack project that includes a website and maybe an app for different platforms, how do you manage notifications (push notifications on web, android and ios, email notifications sms notifications, maybe even telegram notifications...)? Hack yourself or do you have a library to handle that task?

How we built a student project platform using Graphql, React, Golang, Ory Kratos and Kubernetes, part 3: Authentication

Ole Petersen — Sun, 03 Apr 2022 16:36:31 +0000

In this series, we are exploring the steps we took to build a student project platform where students can share their sideprojects and explore what their peers are doing.
After looking at the architecture of our GraphQL backend and how we consume it in a typesafe way, we will take a deeper dive into authentication.

High-level architecture

We have three software components that are involved in the authentication process:

The React UI. The user enters their password here and wants to see their protected data
The GraphQL API. The API wants to verify that requests that need authentication really come from the user the request claims to come from. Located in our kubernetes cluster
Ory Kratos. Kratos manages user login credentials. Located in our kubernetes cluster, too.

Since I have been confused by it in the beginning, I created an image of all the messages sent in a typical login:

Flows

Ory Kratos often works with so-called flows. A flow is a process that my consist of several steps such as account recovery or login (with MFA in the future). For login with username/password this may seem like an overkill since there is only a singe request (send credentials in exchange for session cookie), but I expect the flow pattern is used here to allow for MFA logins in the future.

So the login is as follows:

Request a login flow
Kratos returns login flow ID
Send login to kratos, referencing the flow ID
Kratos returns session cookie
Request protected field from GQL api, attaching session cookie
GQL API asks kratos to verify session cookie
Kratos returns user ID belonging to session cookie
GQL API returns protected field

Redirects / Account recovery

Sometimes, Kratos and the user interface need another concept to interact. Let's say you forgot your password and requested a recovery link via mail. The problem kratos developers needed to solve here is to send some token from the recovery link back to kratos (to verify the email) and at the same time associate it with a session of the UI in a web browser so that the user can reset their password in the UI, and all with minimal knowledge of the specific app architecture.

They solved this by hosting kratos on the same domain as the app itself and using redirects. For instance our UI is hosted under https://huddle.hsg.fs.tum.de/ui whereas kratos is hosted under https://huddle.hsg.fs.tum.de/.ory/kratos. The messages exchanged are as follows:

User clicks recovery link in the format https://huddle.hsg.fs.tum.de/.ory/kratos/self-service/recovery?flow=6b817dhf-4717-48we-a756-cd7902b43ce6&token=QiGjssJcvB08H7cUdE3tyXvQr1i4OSaR. Note that this will result in a GET request by the browser that will be handled by kratos totally independent from the UI!
Kratos verifies the query parameters and responds with a redirect to a preconfigured URL such as https://huddle.hsg.fs.tum.de/ui/profile. A session cookie is set as well so that future requests by the user can be authenticated
After the redirect, the web application UI takes over. For instance the user can set a new password on the profile site.

Deployment and secret management

Let's first go through the deployment of ory kratos itself, which we do in kubernetes with helm and helmfile.

Kratos has a helm chart that allows to install kratos in a cluster more or less conveniently. The interesting thing is how to configure kratos (set redirect links, email-credentials etc.). We to it by providing values to the helm chart in our helmfile. All this is managed in our deployment repo:

├── cert.yaml
├── database
│   ├── migrate.sh
     ...
├── database.yml
├── deployment.yaml
├── helmfile.yaml
├── kratos.yml
└── userSchema.json
...

The entrypoint for the kratos deployment is helmfile.yaml:

repositories:
...
  - name: ory
    url: https://k8s.ory.sh/helm/charts

releases:
...

  - name: kratos
    chart: ory/kratos
    values:
      - kratos.yml

Note how the configuration itself is loaded from another file, kratos.yml:

kratos:
  config:
    courier:
      smtp:
        from_address: huddle.bot@zohomail.eu
        from_name: Huddle
    identity:
        default_schema_url: base64://ewogICAgIiRpZCI6ICJodHRwczovL2dpdGxhYi5scnouZGUvcHJvamVjdGh1Yi9zY2hlbWEtdXNlci1rcmF0b3MuanNvbiIsCiAgICAiJHNjaGVtYSI6ICJodHRwOi8vanNvbi1zY2hlbWEub3JnL2RyYWZ0LTA3L3NjaGVtYSMiLAogICAgInRpdGxlIjogIlBlcnNvbiIsCiAgICAidHlwZSI6ICJvYmplY3QiLAogICAgInByb3BlcnRpZXMiOiB7CiAgICAgICAgInRyYWl0cyI6IHsKICAgICAgICAgICAgInR5cGUiOiAib2JqZWN0IiwKICAgICAgICAgICAgInByb3BlcnRpZXMiOiB7CiAgICAgICAgICAgICAgICAiZW1haWwiOiB7CiAgICAgICAgICAgICAgICAgICAgInR5cGUiOiAic3RyaW5nIiwKICAgICAgICAgICAgICAgICAgICAiZm9ybWF0IjogImVtYWlsIiwKICAgICAgICAgICAgICAgICAgICAib3J5LnNoL2tyYXRvcyI6IHsKICAgICAgICAgICAgICAgICAgICAgICAgImNyZWRlbnRpYWxzIjogewogICAgICAgICAgICAgICAgICAgICAgICAgICAgInBhc3N3b3JkIjogewogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICJpZGVudGlmaWVyIjogdHJ1ZQogICAgICAgICAgICAgICAgICAgICAgICAgICAgfQogICAgICAgICAgICAgICAgICAgICAgICB9LAogICAgICAgICAgICAgICAgICAgICAgICAidmVyaWZpY2F0aW9uIjogewogICAgICAgICAgICAgICAgICAgICAgICAgICAgInZpYSI6ICJlbWFpbCIKICAgICAgICAgICAgICAgICAgICAgICAgfSwKICAgICAgICAgICAgICAgICAgICAgICAgInJlY292ZXJ5IjogewogICAgICAgICAgICAgICAgICAgICAgICAgICAgInZpYSI6ICJlbWFpbCIKICAgICAgICAgICAgICAgICAgICAgICAgfQogICAgICAgICAgICAgICAgICAgIH0KICAgICAgICAgICAgICAgIH0sCiAgICAgICAgICAgICAgICAidXNlcm5hbWUiOiB7CiAgICAgICAgICAgICAgICAgICAgInR5cGUiOiAic3RyaW5nIiwKICAgICAgICAgICAgICAgICAgICAib3J5LnNoL2tyYXRvcyI6IHsKICAgICAgICAgICAgICAgICAgICAgICAgImNyZWRlbnRpYWxzIjogewogICAgICAgICAgICAgICAgICAgICAgICAgICAgInBhc3N3b3JkIjogewogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICJpZGVudGlmaWVyIjogdHJ1ZQogICAgICAgICAgICAgICAgICAgICAgICAgICAgfQogICAgICAgICAgICAgICAgICAgICAgICB9CiAgICAgICAgICAgICAgICAgICAgfQogICAgICAgICAgICAgICAgfQogICAgICAgICAgICB9CiAgICAgICAgfQogICAgfQp9
        schemas: []
    selfservice:
      default_browser_return_url: https://huddle.hsg.fs.tum.de/ui
      methods:
        link:
          enabled: true
      flows:
        settings:
          ui_url: https://huddle.hsg.fs.tum.de/ui/
        verification:
          ui_url: https://huddle.hsg.fs.tum.de/ui/verification
          enabled: true
        recovery:
          enabled: true
          ui_url: https://huddle.hsg.fs.tum.de/ui/profile
    serve:
      public:
        base_url: https://huddle.hsg.fs.tum.de/.ory/kratos
        cors:
          enabled: true
          allowed_origins:
            - https://huddle.hsg.fs.tum.de
            - https://huddle.ridilla.eu
            - http://localhost:3000
  autoMigrate: true
secret:
  # -- switch to false to prevent creating the secret
  enabled: false
  # -- Provide custom name of existing secret, or custom name of secret to be created
  nameOverride: secret-kratos

This config consists of two parts:

The kratos.config subfields follow the schema for configuring kratos independent from kubernetes, as described here. As you can see we set various fields such as the email address to use for account recovery, the url to redirect to after recovery and the identity schema to use for storing account data in base 64 encoding.
The secret subfields are used for a kubernetes-specific way to provide credentials since we don't want to save them in plain text. We state that kratos should use a secret called secret-kratos to get credentials rather than try to take them from the plain text config. This allows us to create the secret in another, encrypted repo (meaning only encrypted versions of the secrets are tracked with git). The secret-kratos-secret has to be in the following format:

kind: Secret
metadata:
  name: secret-kratos
type: Opaque
apiVersion: v1
stringData:
  dsn: postgres://kratos:cmoaincsieadfo@psql-postgresql:5432/kratos
  secretsCipher: nasdlifuhlaiwd
  secretsCookie: lnaslicWERUOZSdicbl
  secretsDefault: ANSIULccbASKNXKAJ
  smtpConnectionURI: smtps://huddle.bot@zohomail.eu:qnIAsdHmsuU@smtp.zoho.eu:465

(of course I replaced the credentials by hitting the keyboard as wild 😃)

Now running helmfile sync will deploy kratos to our cluster!

Finally we need to publicly expose kratos using an ingress controller in kubernetes:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: kratos-ingress
  annotations:
    kubernetes.io/ingress.class: "nginx"   
    nginx.ingress.kubernetes.io/rewrite-target: /$2
spec:
  tls:
  - hosts:
      - huddle.ridilla.eu
    secretName: huddle-ridilla-eu-prod-tls
  - hosts:
      - huddle.hsg.fs.tum.de
    secretName: huddle-hsg-fs-tum-de-prod-tls
  rules:
  - host: huddle.hsg.fs.tum.de
    http:
      paths:
      - path: /.ory/kratos(/|$)(.*)
        pathType: Prefix
        backend:
          service:
            name: kratos-public
            port:
              number: 80

However two things feel a bit hacky about the deployment:

Even though we correctly set the base URL in the kratos config, we need to do regex magic to rewrite the path as if kratos was hosted under the root path.
I was unable to find a way to put the kratos config in its own file (which would for instance enable intellisense in VSCode) and include that file as a subfield of the kratos.config field of the helm config
Running helmfile sync after altering some kratos preferences does NOT deploy these changes to the cluster. You need to delete the kratos deployment and then run the sync comment.

Do you have any tips on how to solve this?

The login UI

Now let's jump right into the code! We handle login requests with a popup component that looks like this:

The component receives an observable of so-called "login request" as a property. By default it's hidden. Whenever a login requests drops in (since a user clicked an item that needs authorization), the component pops up and asks the user to fill in their credentials, register or reset their password. When the credentials are entered, the corresponding flow is started using the kratos js sdk.

import React, { useEffect, useState } from 'react';
import type { Identity, UiContainer, V0alpha2Api } from '@ory/kratos-client'
let _api: V0alpha2Api | undefined
export const getAPI = async () => {
    // allow for code splitting since the kratos sdk is large
    if (!_api) {
        const { V0alpha2Api, Configuration } = await import("@ory/kratos-client")
        const kratosConfig = new Configuration({
            basePath: clientConfig.kratosUrl,
            baseOptions: {
                withCredentials: true
            }
        });
        _api = new V0alpha2Api(kratosConfig);
    }
    return _api
}
import './AuthenticationManagerPopup.css'
import { AuthenticationRequest } from './authenticationObserbale';
import { Observable, useApolloClient } from '@apollo/client';
import Button from '../shared/Button';
import Input from '../shared/Input';
import { clientConfig } from "../config"
import { Link } from 'react-router-dom';
import { useGetMeWithoutLoginPromptQuery } from '../schemas';
export const AuthenticationManagerPopup: React.FC<{ a: Observable<AuthenticationRequest> }> = (props) => {
    const meData = useGetMeWithoutLoginPromptQuery() // This should actually never be needed when the popup is shown
    const client = useApolloClient()
    const [authRequests, setAuthRequests] = useState<AuthenticationRequest[]>([])
    const [email, setEmail] = useState("")
    const [password, setPassword] = useState("")
    useEffect(() => {
        // listen for authentication requests on component creation...
        const subscription = props.a.subscribe(req => {
            setAuthRequests([...authRequests, req])
        })
        // ... and clean up when the component is unmounted
        return () => {
            subscription.unsubscribe()
        }
    }, [])
    if (authRequests.length === 0) {
        return null
    }
    return (
        <div id="authentication-manager-popup">
            <div>
                <h1>Authentication Panel</h1>
                {meData.data?.meIfLoggedIn && <p>Hello {meData.data.meIfLoggedIn?.username}, old friend!</p>}
                <br />
                <Input onChange={setEmail} description='email' autoComplete='email' />
                <br />
                <Input onChange={setPassword} description='Password' autoComplete='current-password' type='password' />
                <br />
                <Button filled disabledMessage={email.length === 0 || password.length === 0 ? "Please enter your email address and desired password above to register" : undefined}
                    onClick={async () => {
                        try {
                            console.log('registering user');
                            const api = await getAPI()
                            const resp = await api.initializeSelfServiceRegistrationFlowForBrowsers()
                            console.log(resp.data);
                            if (!resp) return
                            const registrationFlowId = resp.data.id;
                            const csrf_token = (resp.data.ui.nodes[0].attributes as { value: string }).value;

                            const registrationResponse = await api.submitSelfServiceRegistrationFlow(registrationFlowId, {
                                method: "password",
                                password: password,
                                traits: {
                                    email: email
                                },
                                csrf_token: csrf_token
                            })
                            console.log(registrationResponse.data);
                            resp.data.ui
                            api.initializeSelfServiceVerificationFlowForBrowsers().then(resp => {
                                console.log(resp.data);
                                const csrf_token = (resp.data.ui.nodes[0].attributes as { value: string }).value;
                                api.submitSelfServiceVerificationFlow(resp.data.id, undefined, {
                                    method: "link",
                                    email: email,
                                    csrf_token
                                }).then(resp => {
                                    console.log(resp.data);
                                    client.refetchQueries({
                                        include: "active",
                                    });
                                })
                            })
                        } catch (e) {
                            alertUnknownError(e)
                        }
                    }}>register</Button>
                <p>or</p>
                <br />
                <Button filled disabledMessage={email.length === 0 || password.length === 0 ? "Please enter your email address and password above to log in" : undefined}
                    onClick={async () => {
                        try {
                            console.log('login');
                            const api = await getAPI()
                            const resp = await api.initializeSelfServiceLoginFlowForBrowsers()
                            console.log(resp.data);
                            if (!resp) return
                            const csrf_token = (resp.data.ui.nodes[0].attributes as { value: string }).value;

                            const loginResponse = await api.submitSelfServiceLoginFlow(resp.data.id, undefined, {
                                method: "password",
                                password: password,
                                password_identifier: email,
                                csrf_token: csrf_token
                            })
                            console.log(loginResponse.data);
                            authRequests.forEach(r => r.onFinished())
                            setAuthRequests([])
                            client.refetchQueries({ include: "all" })
                        } catch (error) {
                            alertUnknownError(error)
                        }
                    }}>login</Button>
                <p>By registering, you consent to the <Link target="_blank" to="/about">data we collect and our terms of use</Link>. </p>
                <Button disabledMessage={email.length === 0 ? "Please enter your email address above to recover your account" : undefined}
                    onClick={async () => {
                        try {
                            console.log('trying to recover account');
                            const api = await getAPI()
                            const resp = await api.initializeSelfServiceRecoveryFlowForBrowsers()
                            console.log(resp.data);
                            if (!resp) return
                            const loginFlowId = resp.data.id;
                            const csrf_token = (resp.data.ui.nodes[0].attributes as { value: string }).value;

                            const recoveryResponse = await api.submitSelfServiceRecoveryFlow(resp.data.id, undefined, {
                                method: "link",
                                email,
                                csrf_token
                            })
                            console.log(recoveryResponse.data);
                            authRequests.forEach(r => r.onFinished())
                            setAuthRequests([])
                        } catch (error) {
                            alertUnknownError(error)
                        }
                    }}>reset password</Button>
                <br />
                <Button onClick={() => {
                    authRequests.forEach(r => r.onFailed("User closed window"))
                    setAuthRequests([])
                }}>close</Button>
            </div>
        </div>
    )
}

export const logout = async () => {
    console.log('logout');
    const api = await getAPI()
    const resp = await api.createSelfServiceLogoutFlowUrlForBrowsers()
    console.log(resp.data);
    if (!resp) return
    const logoutResponse = await api.submitSelfServiceLogoutFlow(resp.data.logout_token)
    console.log(logoutResponse.data);
}
export const LogoutButton: React.FC = props => {
    return (
        <Button onClick={logout}>Logout</Button>
    )
}
...

Is there a nicer way to handle the CSRF token? This feels a bit hacky...

Right now, the flows can be finished without any further human interventions since they consist only of one step, but this may change with 2FA.

The missing part that I find really cool is how the popup is triggered. We do this by a clever configuration of the Apollo client with Links:

import { HttpLink, ApolloClient, InMemoryCache, ApolloLink, Observable } from "@apollo/client";
import { offsetLimitPagination } from "@apollo/client/utilities";
import PushStream from "zen-push"
import { AuthenticationRequest } from "./authentication/authenticationObserbale";
import {clientConfig}  from "./config";
import { StrictTypedTypePolicies } from "./schemas";

const endpointLink = new HttpLink({
    uri: clientConfig.gqlUrl,
    credentials: "include"
});
// Subscribe to this to display the authentication popup when needing authentication
export const authenticationStream = new PushStream<AuthenticationRequest>();
const LoginLink = new ApolloLink((operation, forward) => {
    const observable = endpointLink.request(operation);
    let waitingForLogin = false
    return new Observable((observer) => {
        // subscribe to http link and handle authentication errors seperately
        const subs = observable!.subscribe({
            next: (data) => {
                if (data.errors?.[0]?.message.includes("authenticate")) {
                    waitingForLogin = true
                    // Ask for login
                    authenticationStream.next({
                        onFailed: () => {
                            observer.next(data)
                            waitingForLogin = false
                        },
                        onFinished: () => {
                            // retry fetching data after authentication
                            subs.unsubscribe()
                            endpointLink.request(operation)?.subscribe({
                                next: data => observer.next(data),
                                complete: () => observer.complete(),
                                error: (err) => observer.error(err)
                            })
                        }
                    })
                } else {
                    // forward data
                    observer.next(data);
                }
            },
            error: (error) => {
                observer.error(error);
            }
            ,
            complete: () => {
                if (!waitingForLogin) {
                    observer.complete();
                }
            }
        });
    });
});
const typePolicies: StrictTypedTypePolicies={
    Project:{
        keyFields:["id"],
        merge(existing, incoming) {
            return { ...existing, ...incoming };
        }
    },
    Query:{
        fields:{
            searchProjects:offsetLimitPagination(["options","searchString"])
        }
    }
}
export const client = new ApolloClient({
    // uri: 'https://huddle.ridilla.eu/api/query',
    cache: new InMemoryCache({typePolicies}),
    link: LoginLink
});

We hook into the http requests sent to our API and whenever an error contains the word "authentication", we ask our user to log in by pushing in the authenticationStream, which is passed to our instance of the AuthenticationPopup component.

The nice thing is that in the rest of the UI, we have no other point, where we need to care about authentication! Just make your GraphQL request and authentication will be handled by the client!

Verifying users in the API backend

Now imagine a user wants to access a protected field of our api (see here for a post about our API architecture) and the backend needs to decide somehow who the requester is in order to know if they have the permission to access the field. On a high level, the backend must extract the session cookie from the request and ask kratos who that cookie belongs to. This is done by injecting some code at the root of our server with the kratos go sdk. The go specific part of the sdk is not documented at all as far as I know so we had to fiddle around with it until it worked.

For every request,we attach the user ID of the requester to the context. But to find out the user ID, we extract the session cookie from the request and check it against kratos with the ToSession function. Note that we only set the cookie for that request and not some header!

package main

import (
    "context"
    "fmt"
    "log"
    "net/http"
    "os"

    "github.com/99designs/gqlgen/graphql"
    "github.com/99designs/gqlgen/graphql/handler"
    "github.com/99designs/gqlgen/graphql/playground"
    kc "github.com/ory/kratos-client-go"
    "github.com/rs/cors"
    "gitlab.lrz.de/projecthub/gql-api/auth"
    "gitlab.lrz.de/projecthub/gql-api/graph/generated"
    "gitlab.lrz.de/projecthub/gql-api/graph/resolvers"
)

const port = "8080"

func main() {
    kratosConfig := kc.NewConfiguration()
    kratosConfig.Host = "kratos-public"
    kratosConfig.Scheme = "http"
    api := kc.NewAPIClient(kratosConfig)
    resolver, _ := resolvers.NewResolver(os.ExpandEnv(os.Getenv("DB_CONNECTION_STRING")))

    config := generated.Config{Resolvers: resolver}
    config.Directives.IsLoggedIn = func(ctx context.Context, obj interface{}, next graphql.Resolver) (res interface{}, err error) {
        if _, loginErr := auth.IdentityFromContext(ctx); loginErr == nil {
            return next(ctx)
        } else {
            return nil, fmt.Errorf("authenticate please")
        }
    }
    srv := handler.NewDefaultServer(generated.NewExecutableSchema(config))

    http.Handle("/api", playground.Handler("GraphQL playground", "/api/query"))
    corsHandler := cors.New(cors.Options{AllowedOrigins: []string{"http://localhost:3000","https://huddle-groups.readme.io"}, AllowCredentials: true, OptionsPassthrough: true, AllowedHeaders: []string{"*"}}).Handler(srv)
    http.HandleFunc("/api/query", func(rw http.ResponseWriter, r *http.Request) {
        cookie, err := r.Cookie("ory_kratos_session")
        if err != nil {
            corsHandler.ServeHTTP(rw, r)
            return
        }
        toSessionReq := api.V0alpha2Api.ToSession(context.Background()).Cookie(cookie.String())
        session, _, err := toSessionReq.Execute()
        if err != nil {
            corsHandler.ServeHTTP(rw, r)
            return
        }

        newContext := auth.NewIdentityContext(r.Context(), &session.Identity)
        newRequest := r.WithContext(newContext)
        corsHandler.ServeHTTP(rw, newRequest)
    })

    log.Printf("connect to http://localhost:%s/ for GraphQL playground", port)
    log.Fatal(http.ListenAndServe(":"+port, nil))
}

We create a GraphQL directive to annotate that a user must be logged in to access a field here, too. Like this we can annotate a field with @isLoggedIn in our schema and if the user is not, an error "authenticate please" will be returned, asking the user to log in in the UI as described above.

We wrapped the code to write or read the identity to/from the context in the auth package:

package auth

import (
    "context"
    "fmt"

    kc "github.com/ory/kratos-client-go"
)

type Identity struct {
    kc.Identity
    traits map[string]interface{}
}
type customKey int

var identityKey customKey

func NewIdentityContext(ctx context.Context, identity *kc.Identity) context.Context {
    return context.WithValue(ctx, identityKey, identity)
}
func IdentityFromContext(ctx context.Context) (*Identity, error) {
    identity, ok := ctx.Value(identityKey).(*kc.Identity)
    if ok {
        traits, ok := identity.Traits.(map[string]interface{})
        if ok {
            return &Identity{Identity: *identity, traits: traits}, nil
        }
    }
    return nil, fmt.Errorf("no identity found in context")
}

func (i *Identity) GetTrait(key string) (string, bool) {
    val, ok := i.traits[key].(string)
    return val, ok
}

The traits property of the sdk's identity type is untyped since it depends on the identity schema, so in our own identity type we wrapped it in a map with a custom getter for string fields.

In a resolver, we can access the identity like this:

func (r *mutationResolver) SetMyDescription(ctx context.Context, description string) (bool, error) {
    me, err := auth.IdentityFromContext(ctx)
    if err != nil {
        return false, err
    }
    err = r.queries.SetDescription(context.Background(), sqlc.SetDescriptionParams{Description: description, ID: uuid.MustParse(me.Id)})
    return err == nil, err
}

Pretty elegant, right?

I hope to have given a nice introduction to Ory Kratos!
Please feel free to comment on what we could do more elegantly! Also I'm interested in your solutions for secret management in Kubernetes!

How to load test data in golang

Ole Petersen — Wed, 30 Mar 2022 20:01:38 +0000

Loading data for your tests is always a pain in go since you do not specify paths relative to the test itself, but relative to the entrypoint of the test.
This brings lots of confusion, especially when there is a function loading test data that is shared between multiple tests at different locations: When retrieving the current directory, the results will be different depending on the calling test.

Also, nobody wants to type ../../../../testdata/test1.json

In order to release this pain, this package allowas you to load test data relative to the your go.mod.
Like this you no more need to worry about the entrypoint of your program and concentrate on the logic you want to do:

Example

Consider the structure

- go.mod
- main_test.go
- data
  - test1.json
  - test2.json

Now run go get github.com/peteole/testdata-loader and modify your test:

  package main_test
  import (
    "testing"

    testdataloader "github.com/peteole/testdata-loader"
  )
func TestMain(t *testing.T){
    test1:=testdataloader.GetTestFile("data/test1.json")
    randomTest:=testdataloader.GetRandTestFile("data/test*.json")
    //now do something using your test data
}

As a nice bonus you can directly copy a file's relative path from the VSCode file explorer and insert it to the data loader!

The code is here:
https://github.com/peteole/testdata-loader

How to manage interests on dev.to

Ole Petersen — Wed, 23 Mar 2022 22:00:22 +0000

Hi, I have been wondering gow you can modify the tags you are interested in Forem. I remember entering them during setup and now I can modify their weights, but I do not see a way to add new ones.

Am I blind, is this hidden somewhere or is the final test that you can only change them via a hidden API:) ?

How we built a student project platform using GraphQL, React, Golang, Ory Kratos and Kubernetes, part 2: Typesafe GraphQL client

Ole Petersen — Mon, 14 Mar 2022 18:42:38 +0000

After explaining how we built our student project graphql API in a typesafe way, we will continue by having a look at the client side.

In terms of technology we use React (typescript) with the Apollo GraphQL Client as well as a code generator for type safety.

Apollo client

The Apollo client has some serious advantages:

The whole application state is kept in an advanced cache which requires only minimal configuration. This minimizes network traffic and keeps the UI elements in sync.
Nice integration with React
Well customizable

This is the basic usage:

// main.tsx
import App from './App'
import {
  ApolloProvider,
  ApolloClient
} from "@apollo/client";
export const client = new ApolloClient({
    uri: 'https://huddle.hsg.fs.tum.de/api/query',
    cache: new InMemoryCache(),
});
ReactDOM.render(
  <React.StrictMode>
    <ApolloProvider client={client}> //inject the client here
        <App/>
    </ApolloProvider>
  </React.StrictMode>,
  document.getElementById('root')
)

// App.tsx
import { gql, useQuery } from '@apollo/client';
const App: React.FC = () => {
    const [projectId, setProjectId]=useState("")
    const {data} = useQuery(gql`
        query($id: ID!){
            getProject(id: $id) {
                name
                description
            }            
        }
    `,{variables:{id:projectId}}
    )
    return (
        <div>
            Enter project ID to explore
            <input onChange={(newId)=>{
                setProjectId(newId)
            }}>
            <div>
                <p>Project name: {data.getProject.name}</p>
                <p>Project description: {data.getProject.description}</p>
            </div>
        </div>
    )
}
export default App

This little code will allow you to explore huddle projects!

Introduce typesafety

The code above already looks nice, but the data returned and the variables used in the useQuery are untyped. To fix this issue we will introduce yet another code generator:

With GraphQL Code Generator you define the queries in a document and let the code generator generate typesafe versions of the useQuery apollo hook (using the GraphQL schema of your API).

The setup is simple:

yarn add graphql
yarn add @graphql-codegen/cli
yarn graphql-codegen init
yarn install # install the choose plugins
yarn add @graphql-codegen/typescript-react-query
yarn add @graphql-codegen/typescript
yarn add @graphql-codegen/typescript-operations

Now let's configure the code generator by editing the newly created file codegen.yml:

overwrite: true
schema: https://huddle.hsg.fs.tum.de/api/query # link your API schema here
documents: operations/* #define graphql queries you want to use react here
generates:
  src/schemas.ts: #the generated code will end up here
    plugins:
      - "typescript"
      - "typescript-operations"
      - "typescript-react-apollo"
      - typescript-apollo-client-helpers

You can now add operations you want to use in your components in operations/projectOperations.gql:

query getProjectById($id: ID!) {
  getProject(id: $id) {
    id
    name
    description
    creator {
      username
      id
    }
    location {
      name
    }
    saved
    tags
...
  }
}

Installing the GraphQL VSCode extension and creating the graphql.config.yml file with the following content

schema:
  - https://huddle.hsg.fs.tum.de/api/query
documents: ./operations/*.graphqls

will even give you intellisense in the operations

Executing yarn run graphql-codegen will do all the magic for you!
Let's say we want to implement the ProjectDetail-component which displays details of the project with the id passed in the props. We can now import the useGetProjectByIdQuery hook!

import { useGetProjectByIdQuery, ...} from '../schemas';
import { ImageGallery } from '../shared/ImageGallery';
import ReactMarkdown from 'react-markdown';
...
export type ProjectDetailProps = {
    id: string
    onBackClicked?: () => void
}
const ProjectDetail: React.FC<ProjectDetailProps> = (props) => {
    const projectResult = useGetProjectByIdQuery({ variables: { id: props.id } });
 ...
    if (props.id == "") return <div></div>
    if (projectResult.loading) return <div className='project-detail'>Loading...</div>
    if (projectResult.error) return <div className='project-detail'>Error: {projectResult.error.message}</div>
    const images = projectResult.data?.getProject?.images
    return (
        <div className="project-detail">
...
            <h1>{projectResult.data?.getProject?.name}</h1>
...
            <ReactMarkdown >{projectResult.data?.getProject?.description || "(no description provided)"}</ReactMarkdown>
            {images && images.length > 0 ? <div >
                <ImageGallery images={images.map(image => ({
                    url: image.url,
                    description: image.description || undefined
                }))} />
            </div> : null}
            <p>Created by {projectResult.data?.getProject?.creator.username}</p>
...
        </div>
    );
}

export default ProjectDetail;

Note that this hook is fully typed:

Nice! It's this easy to make an API end-to-end typesafe!

Now as a bonus let's have a look at how to customize the cache to our needs.
Let's say we update a project at some place in the code. We want Apollo to sync the update to all the components we used in the code. To do so, we need to tell Apollo somehow to decide which Project objects correspond to the same object (and must therefore be updated) and how to apply updates to the cache for instance if only a few fields are refetched with a new value. This is done by passing a TypePolicies object to the Apollo client cache. The type of this object is also generated by our code generator. So let's do it:

// main.tsx
import App from './App'
import { StrictTypedTypePolicies } from "./schemas";
import { offsetLimitPagination } from "@apollo/client/utilities";
import {
  ApolloProvider,
  ApolloClient
} from "@apollo/client";
const typePolicies: StrictTypedTypePolicies={
    Project:{
        keyFields:["id"], // treat Project objects with the same id as the same project
        merge(existing, incoming) { //merge new projects on old projects. This may be more advanced.
            return { ...existing, ...incoming };
        }
    },
     Query:{
        fields:{
            searchProjects: offsetLimitPagination()
        }
    }
}
export const client = new ApolloClient({
    uri: 'https://huddle.hsg.fs.tum.de/api/query',
    cache: new InMemoryCache({typePolicies}),
});
ReactDOM.render(
  <React.StrictMode>
    <ApolloProvider client={client}> //inject the client here
        <App/>
    </ApolloProvider>
  </React.StrictMode>,
  document.getElementById('root')
)

The custom merge function can also be used to concatenate parts of an infinite feed of results to one list. Since the query uses "offset" and "limit" as parameters, we can use the existing merger function offsetLimitPagination provided by Apollo, which merges results by concatenating the result lists according to the offset and limit parameters.
Like this you can trigger a fetching of more results and append them to current result list flawlessly when the user scrolls towards the end of the list.

For instance we have a searchProject function which receives an offset and a limit of results. This is how we implement an infinite scroll bar:

//HomePage.tsx
import { useRef, useState } from 'react';
import HomeHeader from '../home-header/home-header';
import ProjectList from '../project-list/project-list';
import { useSearchProjectsQuery } from '../schemas';
import "./home-page.css"

function HomePage() {
    const [searchString, setSearchString] = useState("");
...
    const projectData = useSearchProjectsQuery({ variables: { searchString: searchString, limit: 10, options: getOptions(category) } })
    const lastRefetchOffset = useRef(-1)// keep track of the last offset we refetched to see if currently new data is loading already
    const onScrollToBottom = () => {
        if (lastRefetchOffset.current === projectData.data?.searchProjects?.length) {
            return;// already loading, so do nothing
        }
        lastRefetchOffset.current = projectData.data?.searchProjects?.length || -1;
        projectData.fetchMore({
            variables: {
                offset: projectData.data?.searchProjects?.length,
                limit: 10,
                options: getOptions(category),
                searchString: searchString
            }
        })
    }
    const entries = projectData.data?.searchProjects.map(p => ({
        description: p.description,
        id: p.id,
        name: p.name,
        ...)) || []
    return (
        <div style={{ position: "relative" }}>
            <HomeHeader onSearchStringChange={(searchString: string) => {
                setSearchString(searchString) // HomeHeader contains a search bar whose updates we can subscribe to here
            }} .../>
            <div className='home-bottom'>
                <ProjectList entries={entries} onScrollToBottom={onScrollToBottom} />
            </div>
        </div>
    );
}

export default HomePage;

I hope you liked this collection of useful tips for using GraphQL on the client side. Feel free to comment!

Stay tuned for the next part where I will discuss how we handele authentication with Ory Kratos!

How we built a student project platform using Graphql, React, Golang, Ory Kratos and Kubernetes, part 1

Ole Petersen — Thu, 10 Feb 2022 19:15:10 +0000

The idea

I started my studies at university in times of Covid. With limited contact to other students, but lots of ideas for side projects, I often dreamed of a platform to share such project ideas on and check out what existing student projects do (it turned out there are actually pretty many of them who do really cool stuff but nobody knows about).
This is how it looks like:

If you want just check out our current prototype.

Our architecture

For scalability and easy deployment (and also just because we can) we decided to deploy all our code in a Kubernetes cluster. For development we do only need little resources, so we just rented a 3-Dollar-a-month VM with a public IP and installed k3s on it.

We exchange most data using a Graphql API that is served by a Golang-Application. We use a schema-first-approach, i.e. the source of truth for what our API can do is a graphql schema. From that schema, we generate both typesafe client- and server code.
Authentication is handled by ory kratos.

The UI is built with React and Apollo Client.

As a database, we use an in-cluster postgresql instance.

The API

First of all, you can play around with our API here and find the code here
Our API is built with gqlgen. The folder structure looks as follows:

...
├── go.mod
├── go.sum
├── gqlgen.yml # config for auto-generating go-code from gql-schema
├── graph
│   ├── generated
│   │   └── generated.go
│   ├── model # partly auto-generated, partly manually edited representations of the graphql datastructures
│   │   ├── models_gen.go
│   │   └── user.go
...
│   ├── resolvers # The code that actually handles graphql requests,  method heads are auto-generated from schema
│   │   └── user.resolvers.go
...
│   └── schema
│       └── user.graphqls
...
├── server.go # entrypoint
├── sqlc # generated database query code
│   └── users.sql.go
...
├── sqlc.yaml # config for autogenerating go-code for sql queries
├── sql-queries # queries we want go-code for
│   └── users.sql
...
└── tools.go

You can initialize most of this project structure quickly by following this comprehensive guide.

Now implementing new features for our API is a joy! The workflow is the following:

Add the new feature to our graphql schema. Say for instance we want to enable our API to add numbers. We create a file called adder.graphqls (in the schemas folder) with the following content:

extend type Query{
    addNumber(a:Int!,b:Int!):Int!
}

Run the codegen comand:

go run github.com/99designs/gqlgen generate

A new file graph/resolvers/adder.resolver.go will be created with the following content:

package resolvers

// This file will be automatically regenerated based on the schema, any resolver implementations
// will be copied through when generating and any unknown code will be moved to the end.

import (
    "context"
    "fmt"

    "gitlab.lrz.de/projecthub/gql-api/graph/generated"
)

func (r *queryResolver) AddNumber(ctx context.Context, a int, b int) (*int, error) {
    panic(fmt.Errorf("not implemented"))
}

// Query returns generated.QueryResolver implementation.
func (r *Resolver) Query() generated.QueryResolver { return &queryResolver{r} }

type queryResolver struct{ *Resolver }

All we have to do now is implement the method:

package resolvers

// This file will be automatically regenerated based on the schema, any resolver implementations
// will be copied through when generating and any unknown code will be moved to the end.

import (
    "context"
    "fmt"

    "gitlab.lrz.de/projecthub/gql-api/graph/generated"
)

func (r *queryResolver) AddNumber(ctx context.Context, a int, b int) (int, error) {
    return a+b,nil
}

// Query returns generated.QueryResolver implementation.
func (r *Resolver) Query() generated.QueryResolver { return &queryResolver{r} }

type queryResolver struct{ *Resolver }

See how we get perfectly typesafe code here!

With this little setup, we are able to run our server and get documentation for free!

Now let's look at how we actually serve useful data with database queries. Take for instance our API for getting a project by its ID:

# project.graphqls

type Project {
  id: ID!
  name: String!
  description: String!
  languages: [String!]!
  location: Location
  participants: [User!]!
  creator: User!
  images: [Image!]!
  createdAt: Time
  # if the current user saved this project
  saved: Boolean!
  tags: [String!]!
}

extend type Query {
  getProject(id: ID!): Project
}

The generated go function head looks like this:

func (r *queryResolver) GetProject(ctx context.Context, id string) (*model.Project, error)

Now we created an SQL query in the file sql-queries/projects.sql:

-- name: GetProjectByID :one
SELECT *
FROM projects
WHERE id = $1;

We now use sqlc to generate typesafe go code for this query. To do so, we need the current database schema, so we created a nice little script that port-forwards our database from the cluster, dumps out the schema and invokes sqlc:

export POSTGRES_PASSWORD=$(kubectl get secret --namespace default psql-postgresql -o jsonpath="{.data.postgresql-password}" | base64 --decode)
kubectl port-forward --namespace default svc/psql-postgres 5432:5432 &
sleep 2
PGPASSWORD="$POSTGRES_PASSWORD" pg_dump --host 127.0.0.1 -U postgres -d postgres -p 5432 -s > schema.sql
rm -Rf sqlc
sqlc generate
kill $(jobs -p)

sqlc is configured to output the queries in the sqlc-subfolder:

# sqlc.yaml
version: "1"
packages:
  - path: "sqlc"
    name: "sqlc"
    engine: "postgresql"
    schema: "schema.sql"
    queries: "sql-queries"

So now we can inject the database code into our resolver:

// resolvers/resolver.go
package resolvers

import (
    "database/sql"

    "gitlab.lrz.de/projecthub/gql-api/sqlc"

)

// It serves as dependency injection for your app, add any dependencies you require here.

type Resolver struct {
    queries *sqlc.Queries
}

func NewResolver(connstring string) (*Resolver, error) {
    db, err := sql.Open("postgres", connstring)
    if err != nil {
        return nil, err
    }
    queries := sqlc.New(db)
    return &Resolver{
        queries: queries,
    }, nil
}

This allows us to make database queries in every resolver function, so let's apply this to our project-by-id-resolver:

func (r *queryResolver) GetProject(ctx context.Context, id string) (*model.Project, error) {
    dbProject, err := r.queries.GetProjectByID(context.Background(), uuid.MustParse(id))
    if err != nil {
        return nil, err
    }
// now just transform the db result to our gql project datatype
    return  return &Project{
        ID:          dbProject.ID.String(),
        Name:        dbProject.Name,
        Description: dbProject.Description,
        CreatorID:   dbProject.Creator.String(),
        Languages:   []string{},
    }, nil
}

Here the auto-generated datatype of the project returned by the db query looks pretty friendly:

package sqlc
type Project struct {
    ID          uuid.UUID
    Name        string
    Description string
    CreatedAt   sql.NullTime
    Creator     uuid.UUID
    Location    sql.NullString
}

Yay!

In the next part, I will discuss how we use our API on our React UI in a typesafe way.

Feel free to comment, ask for details and stay tuned!