DEV Community: Wesley Skeen

A workflow pattern in F#

Wesley Skeen — Tue, 05 Dec 2023 09:37:09 +0000

WesleySkeen / fsharp-workflow-patterns

During a lift, refactor, shift (C# -> F#), I came across a feature in our codebase that was a good candidate for a workflow pattern. It contained the following steps and was fairly procedural in design. Basically the code done some validation and if the validation was successful we would call some external service. Each service had its own steps for validation. There was a lot of mutation and which resulted in huge side effects that made the code hard to debug and read. The layout was something like

var canCallServiceA = false
var canCallServiceB = false
var canCallServiceC = false

if email is poulated and phone number is populated
    canCallServiceA = true
else
    canCallServiceA = false

if phone number is populated and canCallServiceA = false
    canCallServiceB = true
else
    canCallServiceB = false

if phone number is not populated and canCallServiceB = true
    canCallServiceC = false
else
    canCallServiceC = true

This is a really simplistic version of what is actually going on in the code base.

Since we didn't want to straight up migrate and inherit the design of the current implementation, we came up with a pattern to convert the logic into a clean workflow pattern.

Seq functions

We heavily relied on Seq functions to get this one across the line, in particular Seq.Fold. This allows us to update state while executing the workflow. Here is basic example of a workflow to demonstrate Seq.Fold.

let shouldContinueWorkflow (input: int) = input <> 8

let executeWorkflow =
    let inputs = [1; 2; 3; 4; 5; 6; 7; 8; 9; 10]
    let shouldExecuteCurrentStep = true

    inputs
        |> Seq.fold (fun (shouldExecuteCurrentStep: bool) (x: int) ->

            match shouldExecuteCurrentStep with
            | true ->
                let shouldContinue = x |> shouldContinueWorkflow

                match shouldContinue with
                | true ->
                    printfn $"Continuing from input is {x}"
                | false ->
                    printfn $"Should not continue as input is {x}"

                shouldContinue
            | false -> false
        ) shouldExecuteCurrentStep

In the above example,

Iterate over the input of an array of integers [1...10]
Check if we should continue (defaulted to true)
Check if the current iteration of input is not equal to 8
Set the state variable shouldExecuteCurrentStep to the result of this check

If we run the above program we get the output

Continuing as input is 1
Continuing as input is 2
Continuing as input is 3
Continuing as input is 4
Continuing as input is 5
Continuing as input is 6
Continuing as input is 7
Should not continue as input is 8

This is a basic example but next we will go into something that matches task more closely

Detailed workflow example

Models

We are going to use more complex objects for

Passing in request data to the workflow functions
Returning data from the workflow validation
What we pass into the workflow to iterate over

Request

In a real world scenario we would be dealing with real data and not an array of 1 to 10. To make this example seem closer to a real work scenario, I am going to imagine the a request will come in that contains real world data. This is the data we will use to decide if a service is called and if we should continue

type ExternalServiceRequest = {
    EmailAddress: string
    PhoneNumber: string
    PostCode: string
}

Result from the validation step

This will hold the state data from the validation step that gets executed on each iteration

type ExternalServiceCanExecuteResult = {
    CanExecute: bool
    ShouldContinue: bool
}

CanExecute - Can this service be called
ShouldContinue - Should we continue with other workflow steps after calling the service

External service

These models take the input of ExternalServiceRequest and have 2 functions

CanExecute - Take ExternalServiceRequest and decide if we should call the service
Execute - Call the service. For now we are just mocking the result of this

Since we have several services that we want to call, it would make sense that these services would inherit from some sort of base class. I used OO to achieve this and created an abstract class.

type IExternalService =
    abstract member CanExecute : ExternalServiceRequest -> ExternalServiceCanExecuteResult
    abstract member Execute : ExternalServiceRequest -> unit

Here are a couple of implementations of this abstract class. Please see the attached git repo to view all the implementations

let serviceA = {
    new IExternalService with                            
        member _.CanExecute request: ExternalServiceCanExecuteResult =
              match
                  request.EmailAddress.Length > 0,
                  request.PhoneNumber.Length > 0,
                  request.PostCode.Length > 0 with                
              | true, true, true -> { CanExecute = true; ShouldContinue =  false }
              | _, _, _ -> { CanExecute = false; ShouldContinue =  true }

        member _.Execute request = printfn "Executing service A"
}

Each service implements the abstract class and it's functions.

Taking serviceA as an example as its CanExecute function is more complex. This fictional service requires email, phone and postcode to be populated in order to execute. If all are populated, then we can call this service and there is no need to call any other service as we get all the required info from this service. If any of the fields are not populated then we do not call this service, but we can continue to call the next service in the workflow

let serviceB = {
        new IExternalService with                            
            member _.CanExecute request: ExternalServiceCanExecuteResult =
                  match request.PostCode.Length > 0 with
                  | true -> { CanExecute = true; ShouldContinue =  true }
                  | false -> { CanExecute = false; ShouldContinue =  true }                 

            member _.Execute request = printfn "Executing service B"
    }

serviceB is another example, but this only requires postcode to be populated. If postcode is populated then we can call this service, otherwise we continue the workflow.

Here are the rest of the implementations

let serviceC = {
    new IExternalService with                            
        member _.CanExecute request: ExternalServiceCanExecuteResult =
              match request.EmailAddress.Length > 0 with                
              | true -> { CanExecute = true; ShouldContinue =  true }
              | false -> { CanExecute = false; ShouldContinue =  true }

        member _.Execute request = printfn "Executing service C"
}

let serviceD = {
    new IExternalService with                            
        member _.CanExecute request: ExternalServiceCanExecuteResult =
              match request.PhoneNumber.Length > 0 with                
              | true -> { CanExecute = true; ShouldContinue =  true }
              | false -> { CanExecute = false; ShouldContinue =  true }

        member _.Execute request = printfn "Executing service D"
}

Executing the workflow

First we have our collection of services we want to call

let externalServices: IExternalService seq =
    [
        serviceA
        serviceB
        serviceC
        serviceD
    ]

Here is the full workflow execution using the above services collection

let executeWorkflow
    (request: ExternalServiceRequest) =

        let shouldExecuteCurrentStep = true

        externalServices
            |> Seq.fold (fun (shouldExecuteCurrentStep: bool) (externalService: IExternalService) ->

                match shouldExecuteCurrentStep with
                | true ->
                    let result = request |> externalService.CanExecute

                    match result.CanExecute with
                    | true -> externalService.Execute request
                    | false -> printfn $"Skipping {externalService.GetType().Name}"

                    result.ShouldContinue
                | false -> false
            ) shouldExecuteCurrentStep

Moving to a pattern like this made the code more concise and readable. One of the big things we are striving for is developer experience. A big factor in this is how we write and structure our code. If a developer looks at the codebase we want them to feel comfortable and confident that they can contribute.

Add deployment annotations to grafana dashboards

Wesley Skeen — Fri, 19 May 2023 14:57:59 +0000

When noticing spikes in exceptions on our dashboards, one of the first things we do is check if there has been a recent deployment. Checking to see if a deployment caused the spikes.

This can be straightforward, but if you have many teams deploying many apps all throughout the day, it may be a little trickier tracking down a related deployment.

I am going to talk you through how to add a grafana annotation to visualize when a deployment has taken place. This will be a simple annotation with some tags.

Here is what we are gonna create

First of all, I am going to be running this using my grafana playground which you can find here

Setup Grafana, Jaeger & Zipkin locally

Wesley Skeen ・ Nov 21 '22

#grafana #dotnet

Once you have that set up, there are a few steps we need to do to enable annotations to display on a dashboard.

Create a service account

Browse here and click Add service account. If you have the grafana playground running, you should be able to access the link.

Give it a name and the Editor role

Create a service account token

Click Add service account token

Then click Generate token

Once you have this token, copy it and keep it save.

Create a dashboard

For this example, I am just gonna create a really simple dashboard here with a random metric. Just to have something to show.

Create annotations

Next we will create some annotations to display. We are going to send them as a http post request. The URL we create annotations on is http://localhost:3000/api/annotations.

We next have to add the Authorization header. The value for this will be Bearer {your service account token we created}.

The last thing to do is send a request. The payload will look like



{
  "created": EPOCH with milliseconds, // eg 1684502867000
  "updated": EPOCH with milliseconds,
  "time": EPOCH with milliseconds,
  "timeEnd": EPOCH with milliseconds,
  "text": "Deployment", // text to display on the annotation pop up
  "tags": [ // tags to display on the annotation pop up
    "kind:deployment",
    "project:your-project-or-app-name",
    "env:prod",
    "region:eu-west-1"
  ]
}

The response should be



{
    "id": {annotation_id},
    "message": "Annotation added"
}

Now that we have created an annotation, it is time to set up our dashboard to view them.

Set up dashboard annotations

Go to the dashboard you created previously and go to it's settings.

Next click on Annotations and then Add annotation query.

Chose Grafana as the data source.

Next under Filter by, choose Tags and then turn on Match any.

You can then filter by tags you sent when sending the http request. I am just gonna filter on one tag kind:deployment.

Click Apply and Save dashboard.

Browse back to your dashboard and you now should see an annotation being displayed

If you don't see any annotations check the following things

You are using a recent timestamp for your EPOCH time when creating your annotations
You are filtering by the correct tags
Your dashboard has annotations turned on

Adding to Azure DevOps pipeline

I have created a gist that can be added to your Azure DevOps pipeline.

F# Orleans reminder grain

Wesley Skeen — Fri, 21 Apr 2023 09:04:51 +0000

This is a simple example of how we set up reminder grains in Orleans and f#.

Here's the code so you can follow along

Grain interface

type IYourReminderGrain =
    inherit IGrainWithStringKey
    inherit IRemindable

    abstract member WakeUp : unit -> Task

We have 2 important parts here that will be implemented in the grain.

IRemindable: This forces us to implement the ReceiveReminder function. This is where the logic we want to execute each time the reminder gets triggered.
WakeUp : This activates the grain so we can set up the reminder

Grain implementation

type YourReminderGrain() =
    inherit Grain()
    interface IYourReminderGrain with
        member this.WakeUp() = Task.CompletedTask
        member this.ReceiveReminder(reminderName, status) =
            Console.WriteLine(reminderName, status)
            Task.CompletedTask

    override _.OnActivateAsync(cancellationToken:CancellationToken) = 
        let _periodInSeconds = TimeSpan.FromSeconds 60
        let _dueInSeconds = TimeSpan.FromSeconds 60
        let _reminder = base.RegisterOrUpdateReminder(base.GetPrimaryKeyString(), _dueInSeconds, _periodInSeconds)
        base.OnActivateAsync(cancellationToken)

There are 2 important parts here

Implement the IYourReminderGrain : The WakeUp is simple as we do not want to do anything here. ReceiveReminder contains our logic. What we want to execute on every trigger of the reminder. Here we are simply writing the reminder name to the console.
Overriding OnActivateAsync : This is where we register the reminder and specify at what interval it should trigger at. _periodInSeconds specifies the time between triggers and _dueInSeconds specifies how long to wait until the first trigger

Kicking off the reminder grain

In this example I do this by adding a hosted service GrainActivatorHostedService.

This is responsible for retrieving the reminder grain from the IGrainFctory and calling the WakeUp grain function

let! _ =
    _client
        .GetGrain<IYourReminderGrain>(Guid.NewGuid().ToString())
                    .WakeUp()

That's it. To run the app just run

> sh run-silo-local.sh
or
> sh run-silo-docker.sh

You can also run it in your IDE or VS Code devcontainers

To verify everything is working correctly:

Dashboard: http://localhost:8081
Dashboard if running with Docker http://localhost:3000

This is part of the road-to-orleans project. Check it out

PiotrJustyna / road-to-orleans

This repository illustrates the road to orleans with practical, real-life examples. From most basic, to more advanced techniques.

road to orleans

road to orleans

This repository illustrates the road to orleans with practical, real-life examples such as .NET solutions. From most basic to more advanced techniques. The code is written using .NET 6 and was tested on macOS (Catalina 10.15.7) and, wherever docker is supported, Linux (Alpine 3.12).

build & run

IDE: build + run (first the cluster, then the client)
run-[client/silo]-[docker/local].sh

monitoring

Silo dashboards are available by default on localhost:8080 unless configured otherwise in the code/dockerfile/run-[client/silo]-[docker/local].sh. Additionally, Datadog APM support is illustrated in solution 3b.

code

flowchart TD

Solution1(Solution1: One basic silo, no grains.)

Solution2(Solution2: One basic silo, one grain, one console client.)

Solution3(Solution3: One basic silo, one grain, one console client, <br/>everything containerized.)

Solution3a(Solution3a: Solution3 + grain persistence.)

Solution3b(Solution3b: Solution3 + datadog APM.)

Solution4(Solution4: First in-memory clustering example. <br/>Many silos, many clients.)

Solution5(Solution5: Solution4 where

…

View on GitHub

Detecting PII leakage in logs

Wesley Skeen — Mon, 13 Mar 2023 09:00:29 +0000

First I wanted to mention I collaborated on this project and article with @mereta.

Before we begin, I want to direct you to the post I published to set up grafana locally using docker. Here you will find simple steps to get your environment set up to experiment.

Setup Grafana, Jaeger & Zipkin locally

Wesley Skeen ・ Nov 21 '22

#grafana #dotnet

Once you have this running, I want to direct you towards the promtail.yml file. This is what we are going to change to let promtail apply our PII detection logic.

Pipeline Stages

We are going to add pipeline_stages to this file.

Simply put, each log that gets passed through promtail will go through these stages. We can perform a number of actions that you can read in detail about here in the grafana docs, but I will go through stages to

Detect PII
Validate the result of the detection
Create a label to hold the result

Detect PII

As part of the stages section, we added the regex stage

- regex:
    expression: '(?P<sensitive_email>([a-zA-Z0-9._-]+@[a-zA-Z0-9._-]+\.[a-zA-Z0-9_-]+))'

Here we add an expression. This is built up of 2 parts
(?P<{0}>({1}))

0 - This is the variable that holds the result of the regex match
1 - This is the actual regex used on the log content

Validate the result of the detection

Next we have the template stage

- template:
    source: sensitive_email
    template: '{{ not (empty .Value) }}'

This stage takes the result held in the variable that was set in the regex stage and applies some logic to it. This logic also updates the value of the variable.

Log	value in `sensitive_email`	`{{ not (empty .Value) }}`	`sensitive_email` new value
My email is JP@mail.com	JP@mail.com	true	true
My email is ***		false	false

Create a label to hold the result

For this all we have to do is add the following

- labels:
    sensitive_email:

This adds a label to the log and sets its value to what is held in sensitive_email

Example of it working

I added a log in my API

_logger.LogInformation($"my data is JP@mail.com");

Here is the result in Loki

As you can see, the log line is

and the value of sensitive_email is true

New content of `promtail.yml`

With the above addition of pipeline_stages this file should look like. I have added another example of detecting credit card PII.

server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://loki:3100/loki/api/v1/push

scrape_configs:
  - job_name: system
    pipeline_stages:      
      - match:
          pipeline_name: "security"
          selector: '{app="api"}'
          stages:

            - regex:
                expression: '(?P<sensitive_creditcard>(?:\d[ -]*?){13,16})'
            - regex:
                expression: '(?P<sensitive_email>([a-zA-Z0-9._-]+@[a-zA-Z0-9._-]+\.[a-zA-Z0-9_-]+))'

            - template:
                source: sensitive_creditcard
                template: '{{ not (empty .Value) }}'
            - template:
                source: sensitive_email
                template: '{{ not (empty .Value) }}'            

            - labels:
                sensitive_creditcard:
                sensitive_email:

    static_configs:
      - targets:
          - localhost
        labels:
          job: varlogs
          __path__: /var/log/*local.log
          app: 'api'

Using the results of these stages

There are several things you can do with these new log labels. Among others, you could

Create an alert to detect if PII has leaked into your logs.
Create dashboards to monitor base on the new labels
You can do some interesting things in grafana such as route these logs to a different tenant. This tenant would have special privileges to view logs with PII contained.

Improvements

Merge the results of the regex matches into a single label.

First we need to update the source template to

- template:
    source: sensitive_email
    template: '{{ if not (empty .Value) }} true {{ end }}' 

- template:
     source: sensitive_creditcard
     template: '{{ if not (empty .Value) }} true {{ end }}'

then we add a new source template to merge the results

- template:
     source: sensitive
     template: '{{ or .sensitive_email .sensitive_creditcard false }}'

- labels:
     sensitive:

Setup Grafana, Jaeger & Zipkin locally

Wesley Skeen — Mon, 21 Nov 2022 16:17:59 +0000

Here is an easy way to setup Grafana locally using docker and dotnet.

WesleySkeen / Grafana-Data-Sources

Grafana-Data-Sources

The current set up in this repo is very minimal. I am just touching the surface of what is possible. I wanted to demonstrate how to run the services locally

Build and run the api with docker on local port 4000

> docker build --pull --no-cache -t weather_forecast_api -f ops/docker/Dockerfile .
> docker run -p 4000:5000 --rm weather_forecast_api

You should now be able to view OTEL metrics from the API

> curl http://localhost:4000/metrics

# TYPE process_runtime_dotnet_gc_collections_count counter
# HELP process_runtime_dotnet_gc_collections_count Number of garbage collections that have occurred since process start.
process_runtime_dotnet_gc_collections_count{generation="gen2"} 0 1665568680770
process_runtime_dotnet_gc_collections_count{generation="gen1"} 0 1665568680770
process_runtime_dotnet_gc_collections_count{generation="gen0"} 0 1665568680770

.
.
.

# EOF

Build and run the api with docker compose

Using docker compose, we can combine multiple docker containers into a single deployment These containers can communicate with each other since we can specify what…

View on GitHub

Please refer to the above repo for the source code to run the below commands and follow along.

Build and run the api with docker on local port 4000

> docker build --pull --no-cache -t weather_forecast_api -f ops/docker/Dockerfile .
> docker run -p 4000:5000 --rm weather_forecast_api

You should now be able to view OTEL metrics from the API

> curl http://localhost:4000/metrics

# TYPE process_runtime_dotnet_gc_collections_count counter
# HELP process_runtime_dotnet_gc_collections_count Number of garbage collections that have occurred since process start.
process_runtime_dotnet_gc_collections_count{generation="gen2"} 0 1665568680770
process_runtime_dotnet_gc_collections_count{generation="gen1"} 0 1665568680770
process_runtime_dotnet_gc_collections_count{generation="gen0"} 0 1665568680770

.
.
.

# EOF

Build and run the api with docker compose

Using docker compose, we can combine multiple docker containers into a single deployment.
These containers can communicate with each other since we can specify what network they can be added to.

First build the image

> docker build --pull --no-cache -t weather_forecast_api -f ops/docker/Dockerfile .

Then in the docker folder (ops/docker), run

> docker-compose up

Among other things, this will bring up Promethius. We have set Promethius up to scrape metrics from the /metrics endpoint of the weather forecaset API.
Notice in the promethius.yml file

static_configs:      
      # replace the IP with your local IP for development
      # localhost is not it, as that is w/in the container :)
      - targets: ['weather_forecast_api:5000']

To confirm promethius can access the API metric scraping endpoint, browse to Browse to the targets page. You should see

Notice that the Status is UP

Grafana set up

Browse to the datasources page

username : admin

password : P@ssw0rd

Promethius

Set the URL to http://prometheus:9090 and then click Save & Test

Then go to the explore page and select promethius as the data source. You should be able to access the metrics

Custom metrics

As an example, I have added a custom meter counter that we can use to track the number of requests made to /WeatherForecast

In WeatherForecastController.cs We have

SharedTelemetryUtilities.RequestCounter.Add(1);

This will add a new metric with a name we have specified data.request_counter
We can now view this custom metric in Grafana

Loki

Set the URL to http://loki:3100 and then click Save & Test

Then go to the explore page and select loki as the data source. Under labels, select filename. You should have have a log file accessable.

Traces

In the endpoint /WeatherForecast I added some external calls to show traces in action. Once you make some calls to this endpoint, you will start to generate traces

Zipkin

Browse to zipkin and click RUN QUERY. This should display some traces

Clicking one of these traces will give you more detail. As you can see, there is an over all request and the 2 external calls.

Jaeger

Browse to Jaeger, select a service and click Find Traces. This should display some traces

Similar to Zipkin, clicking on one of the traces will give more detail

DEV Community: Wesley Skeen

A workflow pattern in F#

WesleySkeen / fsharp-workflow-patterns

Seq functions

Detailed workflow example

Models

Request

Result from the validation step

External service

Executing the workflow

Add deployment annotations to grafana dashboards

Setup Grafana, Jaeger & Zipkin locally

Wesley Skeen ・ Nov 21 '22

Create a service account

Create a service account token

Create a dashboard

Create annotations

Set up dashboard annotations

Adding to Azure DevOps pipeline

F# Orleans reminder grain

Grain interface

Grain implementation

Kicking off the reminder grain

PiotrJustyna / road-to-orleans

This repository illustrates the road to orleans with practical, real-life examples. From most basic, to more advanced techniques.

road to orleans

build & run

monitoring

code

Detecting PII leakage in logs

Setup Grafana, Jaeger & Zipkin locally

Wesley Skeen ・ Nov 21 '22

Pipeline Stages

Detect PII

Validate the result of the detection

Create a label to hold the result

Example of it working

New content of promtail.yml

Using the results of these stages

Improvements

Setup Grafana, Jaeger & Zipkin locally

WesleySkeen / Grafana-Data-Sources

Grafana-Data-Sources

Build and run the api with docker on local port 4000

Build and run the api with docker compose

Build and run the api with docker on local port 4000

Build and run the api with docker compose

Grafana set up

Promethius

Custom metrics

Loki

Traces

Zipkin

Jaeger

New content of `promtail.yml`