DEV Community: Elena de Graaf

Feeling tired after work? Maybe that's why

Elena de Graaf — Mon, 26 Feb 2024 18:17:50 +0000

I do not miss the days I spent sitting on the sofa staring at a wall, because this was the only thing I had enough energy for. It seemed impossible to stop scrolling social media and do something for myself. I sleep well, I eat well, I regularly exercise - what else could I possibly do? Turns out, I can do something with my work.

If you prefer a video format and visual explanations, check out the video.

If you are a checklist person, I've got one for you here.

I am a software engineer. My work tasks vary a lot: I spend my time talking to stakeholders to figure out what business requirements are, or planning a new design for the feature or service, or sometimes I even get to code. There's almost never a day where I have one task that I can complete from start to finish, they usually depend on other people, require more information, and span over multiple days. So instead, I jump from one task to another, trying to not forget things and still deliver some quality results. Well, sometimes just results.

And this is extremely, horribly tiring.

So I started trying some tricks and figuring out what works better for me. Depending on how busy my life was getting from day to day, different strategies worked better. Assuming that they don't work for me 100% of the time, there's no guarantee that they work for you, but hey, no harm in trying it out.

Level 1 - Focus on one thing

Applicable on calmer days.

First of all, kill your DMs (direct messages):

Block a slot in a calendar (not even in the future, this one, right now) so your team knows you're busy
Set a status ", expect slow response"
Slow response == no response for X minutes. Turn off the DMs for X minutes, where X is the longest you can go without feeling guilty.

Then make sure you are not distracting yourself during breaks (while waiting for the tests, compilation, etc):

Close your eyes and do nothing (still wish to be able to do that)
Write down some ideas for testing, documentation, etc for the current task
Read a passage from a technical book: small enough to keep your brain engaged, but not too big so you switched the context again.

Level 2 - Assign the environments

Applicable on busy days, but with flexibility, like working from home

use different set of programs/tools for each task
use different styles of music for each task
use different light settings for each task (e.g. desk lamp vs. big light)

The idea is to trick your brain into associating contexts with the environment, to make it easier to switch them.

Level 3 - make notes

Applicable for very busy days

Write down all the active tasks
Write down all the goals for the tasks that you can realistically do within a day
Before you do any small step within a task, write it down
I also use lines to show direction between the steps, it helps me to be less confused
As soon as the step is done, cross it and move on

That's it! I never do all of them in the same day, usually I pick one level based on what I expect the day to be. Let me know if any of these worked for you!

p.s. especially the book part, it's weird, right? I am still surprised that it works for me

How to get the most out of the dev conference? Obviously, host a workshop!

Elena de Graaf — Thu, 03 Aug 2023 19:48:49 +0000

I don't have a fear of public speaking. At least I strongly believed that I didn't - and I finally got a chance to test this theory in practice: I got to participate in the WeAreDevelopers conference with a React server components workshop.
Not that I never tried to present some ideas to a big number of people, but it was usually a part of work or studies, and it was different. Despite the fact that I had to persuade people to listen, I never had to prove myself to anyone.
This time, it was 40 complete strangers with expectations, technology I'm not an expert in, and a complete lack of experience in public speaking. What could go wrong, right?

Preparation

A first couple of practices went relatively well. We agreed on the schema, added some new technologies (but what if I cannot make it work?), worked on the ordering and placement. I even prepared all the necessary code without getting too nervous (where are my logs? ah yeah, on the server, not in the browser console). I learned so much more about React, it was a great opportunity to dig deep.
But then the fun began. Well, then I also broke a finger and continued the whole preparation with one left hand available, but truth to be told, I didn't think much would have changed if I had two hands.
The first run through should have been good. I prepared the talking points, went through them a couple of times, brushed up on the theory… and kind of assumed that all the practical parts will just go very smoothly. I mean, it's just a code, I do it every day, it's not even interesting to practice, right?
Well, I quickly learned that coding in a workshop is a completely separate thing. You cannot run between files like crazy, you have to be very deliberate in everything you do not to confuse people. You have to remember where and what to write. So that required the same amount of practice, if not more.
The second (and final) practice run went better. This time I was prepared for coding, talking points were more or less clear, and we were mostly doing small adjustments and checks. We quickly ran through everything, and were ready.
Almost.

Lessons learned (or at least noticed)

One thing I was prepared for was a feeling of an absolutely empty head right before the workshop. But I've been there before, and that feeling I decided to ignore right away: I've been repeating the same stuff over and over again, and I knew that I was ready, whatever my mind was trying to pull.
But there were things I didn't know about, and I will make sure that I'll take them into account for the next time.
First of all, timing. No, the timing of the workshop itself was surprisingly good, but I was simply not prepared to speak for so long. I was not doing the workshop alone, but we didn't think that my bits would be so long that I will be physically and mentally tired of speaking by the end of the second hour.
The other problem that I did not anticipate was the eye contact with the audience. At work, I usually don't have to think about it because the majority of meetings are online or hybrid, and looking at the screen is a pretty normal thing to do. But there I was torn between looking at my code, looking at the presentation, and looking at people. It is hard enough to connect with a bigger audience, let alone being glued to the screen.
And maybe follow-ups would be nice to have. People spent two hours building something with us, so they should be able to take something from it. A short version of points covered? A small "homework" task they can accomplish in their own time if they want to check if they got the skills?

Conclusion

Even with these learnings, for me, the workshop was amazing. The energy in the room was great: going from a quiet introverted crowd to people communicating, laughing and helping. I think I was extremely lucky to have a great audience.
So yeah, that was something. But I have to say, now I am looking at talks more favourably: it feels like making people listen is easier than making people do something. Maybe I am wrong, and it's way harder - I hope to learn it in practice soon enough.

GitHub Actions: 7 obvious things I missed

Elena de Graaf — Mon, 24 Jul 2023 18:52:37 +0000

GitHub actions are amazing. They are free (ish), fast, powerful — and super frustrating when you’re trying to build them quickly for the first time.

For a long time, my experience with GitHub actions was just about playing around. Recently I had to write a specific workflow, with quickly changing requirements, and that’s where things got tricky.

That experience created some awkward — and sometimes funny — moments that I wanted to share. Some of them are straight up silly, but some may help others to avoid the oops moment of “oh. I just spent 30 minutes on something that is that obvious”.

Even AI generated ducks are less confusing than GitHub context

1. Debugging
Before we dig into anything, the most useful thing that you can do when you are starting to work with GitHub actions is knowing how to see the context of your workflow.

There are different types of context, and you may need different ones depending on your tasks. For example, if you are building retries or conditional steps, it might make sense to have a small output step that will show the outcomes. Might save you a few moments of confusion (see the retries sections for details about said confusion):

- name: Context for steps
       env:
         STEPS_CONTEXT: ${{ toJson(steps) }}
       run: echo "$STEPS_CONTEXT"

2. Mixing conditions
Let’s start with something very simple. The task I had was so straightforward that probably every engineer had had it in their career: to send notifications in case the pipeline fails. To do that, I used a slack GitHub action, and I wanted to utilise it in case the whole workflow fails. There is a very convenient GitHub function for it:

if: ${{ failure() }}

I used the same workflow for pull requests and push to the branch, and that would have been a bit annoying, so I modified the condition to only trigger notifications if the failure is not triggered by a pull request:

if: ${{ failure() }} && {{ github.event != 'pull_request' }}

Some of you may already see the issue, but I was quite surprised to see way more Slack notifications then I anticipated.

Figuring out what was the problem took a minute. Of course, I suspected the condition, but was not sure what was the problem: there were two valid expressions and a logical operator, what could go wrong?

Well… here’s the corrected version:

if: ${{ failure() && github.event != 'pull_request' }}

To use the logical operator, both sides need to be a part of one expression. Another way to avoid that kind of error is to simplify it to:

if: failure() && github.event != 'pull_request'

From GitHub documentation:

When you use expressions in an if conditional, you may omit the expression syntax (${{ }}) because GitHub automatically evaluates the if conditional as an expression.

Overall it was a good reminder to never underestimate the brackets. Ever.

3. Jobs are parallel, steps are not
That sounds obvious to anybody who actually read the whole documentation (anybody else having documentation fatigue? seems like we are using way too many technologies). That was a confusion that you see in practice quickly, so at worst you have one test workflow trying to deploy your code that is not compiled yet.

You can make jobs sequential by references one job from another using needs keyword. You can make steps parallel by placing them into two jobs. Just don’t forget to give all of that decent names or comments to avoid an existential dread of having to make changes to your workflows.

4. Code and names
Why do you need both id and a name for a step? How to set an id for a job? How to reference them? What happens if you don’t?

You will need an id to reference a step. It is a very convenient, but not necessary property. If you don't specify an id, a random id will be assigned to a step.

"1416cd4eabb2812fba6aa416b1a7f858": { <-- Example of random id
    "outputs": {},
    "outcome": "success",
    "conclusion": "success"
  },

To find the step in the UI, you’ll need to use the name property.

steps:
 - name: Step 1 <-- Will be shown on UI
   id: step1    <-- Will be used to reference the step

For jobs, however, the only way to create a job is to give it an id. The purpose of the name is still the same, to display it on UI.

jobs:
  jobId1:         <-- This is an id
  jobId2:
    name: Job 2   <-- You'll see this on UI
    needs: jobId1 <-- Now is it referenced

The-least-pro tip possible: use only underscores _ (or only dashes -) in your workflows for ids and variables. Run edit and replace on your workflows to make sure the other one is not present. Will save you probably a few minutes of staring at a perfectly correct workflow, figuring out what is not working.

It gets worse

5. Sharing is caring, or output from jobs/steps/other workflows
Now that I have a working workflow using all of this, I have no idea why it was so confusing before. But that’s how the brain works, I guess.

To use outputs in steps, your step needs to have an id and be referenced in the context of steps inside the job. Of course, the name of the key should match the key referenced by another step, but at this point it’s pretty obvious:

jobs:
  job1:
    runs-on: ubuntu-latest
    steps:
      - id: step1
        run: echo "stepOutput=value" >> "$GITHUB_OUTPUT"
    steps:
      - id: step2
        run: echo "${{ steps.step1.stepOutput }}"

If you want to reference the step from another job, you have to define the outputs for the job and then use needs context. Now, the name of the key in the step needs to be the same as in the job, but the job output key may be different… another not-so-pro tip: do not make them different unless you absolutely have to. The confusion only grows later:

jobs:
  job1:
    runs-on: ubuntu-latest
    outputs:
      jobOutput: ${{ steps.step1.stepOutput }}
    steps:
      - id: step1
        run: echo "stepOutput=value" >> "$GITHUB_OUTPUT"
  job2:
    runs-on: ubuntu-latest
    needs: job1
    steps:
        run: echo "${{ needs.job1.outputs.jobOutput }}"

And the peak of the sharing of the outputs — reusable workflows. They are especially sweet if you have pretty long workflows, complex file structure and deadlines because the mess of the output names is becoming rather amusing. If you want to use the output from another workflow, you need to define outputs from both the job and the workflow:

name: Reusable workflow

on:
  workflow_call:
    outputs:
      workflowOutput:
        value: ${{ jobs.example_job.outputs.jobOutput }}

jobs:
  example_job:
    runs-on: ubuntu-latest
    outputs:
      jobOutput: ${{ steps.step1.outputs.stepOutput }}
    steps:
      - id: step1
        run: echo "stepOutput=value" >> $GITHUB_OUTPUT

So they can later be used in the main workflow:

name: Call a reusable workflow and use its outputs

on:
  workflow_dispatch:

jobs:
  job1:
    uses: <path>/reusable-workflow-file.yml

  job2:
    runs-on: ubuntu-latest
    needs: job1
    steps:
      - run: echo ${{ needs.job1.outputs.workflowOutput }}

Now imaging combining a few results, add a pinch of - vs _ problems I explained earlier, and that will explain at least a couple of new grey hair.

Oh shoot

6. Third time’s a charm, or retries
Before sharing my journey to this void, let me begin this section with a very nice article explaining different ways to retry a step — if you need a solution, you’ll most likely find it there.

My problem was very specific: there were a number of very unstable approaches, and I had to try each, hoping that at least one succeeds. Not the best way to solve the problems with your software — but good enough for some internal tool.

Each approach was written as a step, each was given a proud if: failure(), and, of course, each didn’t work. If the first approach (let’s call it step1) failed, the whole workflow would fail. Why?

Quick googling gave me continue-on-error: true property, that I added to the first n-1 steps. Without it job is considered failed after any one step has failed. Now the workflow continued working — but step2 and others were still not executed. Why?

Another googling minute, another finding: steps have different parameters for indicating a result of the execution. Using continue-on-error: true results in conclusion being a success in any case, while the outcome can differ. And, as you already guessed, if: failure() looks at conclusion.

"step1": {
    "outputs": {
      "custom_outcome": "value"
    },
    "outcome": "failure",
    "conclusion": "success"
  },

A bit confusing, but ok. Let’s make the check more verbose.

if: ${{ steps.step1.outcome == 'failure' && steps.step2.outcome == 'failure'}}

And yet, there was one problem. Steps can not only fail or succeed, but also be skipped, which was immediately evident when some additional inputs check we added. So the condition for the step3 became this:

if: ${{ steps.step1.outcome != 'success' && steps.step2.outcome != 'success'}}

That was one of the cases where checking the context produced by GitHub was necessary. Without it I was quite confused by all the conclusions, outcomes, outputs, and whatever else I had a misfortune of naming in similar fashion.

7. Default value for the input
There are two ways to define the default value, and of course, I learned the wrong one when I needed it. So, here are the two of them together.

The most obvious way is you can use the default option for input itself:

on:
  workflow_dispatch:
    inputs:
      actor:
        type: string
        default: 'system'

jobs:
  job1:
    runs-on: ubuntu-latest
    steps:
      - run: echo ${{ github.event.inputs.actor }}

Alternatively, you can use the env to define a default value:

on:
  workflow_dispatch:
    inputs:
      actor:
        type: string
        required: false

jobs:
  job1:
    runs-on: ubuntu-latest
    steps:
      - env: ACTOR: ${{ github.event.inputs.actor || 'system' }}
        run: echo $ACTOR

The second option looks more complex, but there are a lot of benefits to it. If you have to get the default value from, say, secrets (encrypted values GitHub stores for you), you will have to use it:

on:
  workflow_dispatch:
    inputs:
      actor:
        type: string
        required: false

jobs:
  job1:
    runs-on: ubuntu-latest
    steps:
      - env: ACTOR: ${{ github.event.inputs.actor || secrets.defaultActor }}
        run: echo $ACTOR

Also, using env is the only way to have a default value if the workflow is used by other triggers than a manual. So even though the first version is simpler, it may make sense to go with the second one anyway, even though it will require to define the default value further from the declaration of the input.

Conclusion

So, those are my notable moments of maybe two or three accumulated days of GitHub actions. I’m quite happy with the tool, I could customise a lot of things to work exactly as I needed, and it gave me so many “oh that’s how it works” moments.

Maybe someday I’ll do a deep dive into them instead of floating on the air mattress on the surface because they definitely deserve it. Or maybe I’ll continue collecting mistakes — at least that make it fun to write about.

How the Miro Developer Platform leverages contract testing

Elena de Graaf — Mon, 15 May 2023 16:12:29 +0000

Design-first approach at Miro

The Miro Developer Platform 2.0 was launched last year, and since then we’ve been working on improving the quality and consistency of our REST API. To achieve this, we decided to embrace a design-first approach. For us, it meant designing the API before actually implementing it, and making sure that the quality was high during the whole development process.

This meant we needed an efficient way to verify compliance between the API and its description in OAS (OpenAPI Specification) files.

In this post, I’ll share why we landed on contract testing as the solution. Contract testing is a fast, lightweight form of API testing that ensures two separate services are compatible and can communicate with each other by checking the consistency with a set of rules.

Switching from code generation

Previously, we generated OAS files from the code, using annotations on controllers and models. At the time, this allowed us to transition from writing the documentation manually to some degree of automation. However, the solution was hacky and required us to jump through a lot of hoops to generate the OAS files.

@Operation(
   summary = "Get specific connector",
   description = "Retrieves information for a specific connector on a board.",
   responses = [
       ApiResponse(
           responseCode = "200",
           description = "Connector retrieved",
           content = [
               Content(schema = Schema(implementation = ConnectorWithLinks::class))
           ]
       )
   ],
   operationId = "get-connector"
)
fun getConnectorById() {...}

The early version of the API was not designed according to OpenAPI standards and had problems like conflicting endpoints and overly complex models. The main components of the design-first approach were quite difficult to implement, so we decided to switch the source of truth from the code-generated OAS files to permanent, curated ones.

How do we keep it in sync?

However, switching the source of truth presented a new problem. When the OAS files were generated from the code, we could at least partially guarantee that we were serving what we were promising. For example, the endpoint description could not have been generated if it didn’t exist in the code. Now, there was no way to tell if the OAS file had the paths and schemas that actually existed.

Looking for a solution

Of course, we have end-to-end tests. Why not reuse them to verify that the API looks as expected? They cover the functionality anyway, so why not add a little schema validation to the mix?

This idea is great, but it has the same flaw as the previous situation — it’s based on trust and leaves room for human error: New endpoint, but tests are not added? New parameter added, but tests are not updated because it’s out of critical path? The whole new experimental API appeared with no new tests and no intention to write them yet?

Writing a set of tests just to verify the compliance between the API and its description in OAS seemed even more far-fetched — not only did it have the same flaws, but it would also require us to do the same work all over again.

And this is how we arrived at the idea of generating tests from the OAS file.

How can a test be generated?

Our OAS has all the necessary data to successfully generate a request and validate the schema of the response. The request body can be built from schemas and the provided example/default values. If we don’t go into the details (and we don’t just yet), the logic for one resource is pretty easy:

First, find the endpoint that creates the resource (for example, the widget).
Save the id from the response.
Re-use it in all other endpoints (get widget by id, update widget).
Find the endpoint to delete it and execute it last.

But the devil is in the details, and the most pressing questions would be: What if it’s more than one resource? How do we guarantee that, say, the widget is created before the board is deleted?

Level-method model

At this point we cared about two things: the method (POST should go first, DELETE — last) and the order of resource creation (boards before widgets). It may seem to have a simple solution.

We can easily identify the endpoints that create/delete the resources by method; that’s the first point of comparison. When we look at the path, we see that some resources appear before others. By naming them resource 1 and resource 2 (3, 4 or however far the craziness can get us), we get the second point. Then we only have to sort in the form of an arrow:

This idea was simple, easy to understand, and worked great as a proof of concept, but it introduced more problems than it solved, namely:

Representing the order in a simple manner possible with two levels of resources, worsens with three levels, and becomes not maintainable after it.
Not all POST endpoints create resources. For example, attaching a tag to the item is also a POST endpoint, but it has to happen after both items and tags are created.
Adding additional order — for example, if the endpoint has parent_item_id, it has to be created after a frame is created — seems impossible without blowing up the complexity.

We needed a more powerful way of sorting the relationships between endpoints, and that led us to a dependency graph.

Dependency graph

Who would have thought to represent dependencies with a graph, right? In any case, it seemed like a very plausible solution: A directed graph can handle relationships of any complexity. It’s very clear what should be executed first. And there’s no need to rely on strict pre-defined ordering.

But how do we build a graph? Do we take the very first endpoint and put it in the graph, then take the next one and try to determine its relationship with the first node? That might work — and guarantee that the board is not deleted before we’re done with widgets on it. But at some point this approach would require traversing the entire graph to figure out the relationships with all other nodes. Despite the fact that we’ll never have that amount of endpoints for it to become a real performance problem, it does not help with the complexity, so we probably need another approach.

Luckily, the level-method ordering approach highlighted not just the problems, but also the way we can use components of the url to determine the relationships. So instead of trying to fit the endpoint into the graph, we’re going to build the graph around it based on what this particular endpoint needs.

Walkthrough

Let’s take a simple endpoint: add a tag to an item. By looking at the endpoint, we can see that before it can be executed, we need to have a board, an item, and a tag already created. We’re going to have a special kind of node — “creational” nodes — and will create a lot of “empty” nodes like this. They can be populated with an actual url template when we get to it, but the node needs to exist before, so it can be used as a dependency.

What are the steps to building a dependency graph for this endpoint?

Create a graph node for current operation.
If the node creates the resource, save it to the map of resources and their corresponding creation nodes.
For each required parameter or resource in the api path:

Find the node that knows how to create the required resource from the map above.
…or create an empty node that will be populated when the creation operation is parsed.
Add a dependency on it.

What if the endpoint deletes the resource entirely? Instead of adding extra complexity to the algorithm, we save them separately, in a “cleanup queue.”

Now we have a graph of all the endpoints, plus a cleanup queue with delete endpoints sorted by the level of the resource being deleted. To test all the endpoints as parameters in the test, we need to put them in a simple sorted list. The initial idea was to use a graph and modified topological search, but with the graph construction that we use, the sorting is simplified to a BFS plus polling the cleanup queue in the end.

As you may have noticed in the above example, an endpoint node can have a dependency on more than one other node, for example, both tags and items. And they, in return, can depend on multiple other nodes. The simple BFS will create duplicates at best, and unexpected ordering in the worst-case scenario. Plus, even duplicates can be detrimental: A second tag is created with the same example value provided in OAS, and this request ends with 400 because duplicated tags are not allowed.

The solution for this, however, would be simple: If some node meets a child with more than one parent, it ignores it and detaches itself from a child, leaving nodes that appear later to take care of. This is the cruel reality of child care in computer science.

Of course, there are still some limitations to what can possibly be automated. Here are some of them:

If the resource cannot be created via API (e.g. organizations for enterprise API), it has to be created before the test runs.
If the endpoint has a parameter (e.g. tag_id or parent_item_id or parent_id), you need to map it to some resource first (e.g. tag_id to tags).
Likewise, if another resource is mentioned in the request body, it has to be created (e.g. widgets for connectors) or mapped to an existing resource.

Customizations

The whole solution consists of 4 parts:

The OAS files describing the Miro API
Test generation logic
Creation of the components that can be generated automatically
The tests themselves

Plugins

The test generation logic is a separate library that only needs to change when a new feature is required for all test suites, for example, when we decide to also make requests with missing required parameters to test error responses. All custom modifications, including providing new OAS files, have to go through plugins.

Plugins offer a set of customizations to the owner of the API. For example, mapping resources to parameters, or retrying logic for some endpoints.

Other types of tests

The initial set of tests contained only positive tests that check that a successful response is returned and the response schema is exactly as it is described in the OAS file. However, there are a number of possibilities to create other types of tests, from the already mentioned missing required parameter checks, to testing the expected error responses.

Automation

The tool is as good as its automation, so the API testing tool is scheduled to run frequently and to notify the team about the results. This solution also unlocked not just publishing the documentation automatically right after the change, but also preparing OAS files, for example, removing endpoints marked as drafts.

There you have it. With OAS contract testing, we can verify compliance between the API and its description in OAS files.

Do you have ideas or a different approach to API testing? Let us know in the comments — we’re interested in what you have to say!

Are you interested in joining our team? Then check out our open positions.

Finally, don’t forget to try the Miro Developer Platform for yourself.