DEV Community: David Rodríguez Donoso

Automating with GitHub issue forms

David Rodríguez Donoso — Mon, 12 Jun 2023 11:40:31 +0000

If you are familiar with issue templates, it is a good way to customise templates whenever anyone wants to create a new issue.

Issue templates need a markdown file, so when a user opens a new issue, it has a template in markdown, so the user needs to modify whatever he wants to have it in the same format.

- What is the bad point about that?
+ A user can modify the template provided, change the headings, or completely remove everything and create a new issue.

A user that creates a new issue with this kind of template can manipulate the issue to change anything that is provided by the template.

New issue forms

So, what are issue forms for?
Issue forms are a new type of issue template. With issue forms, you can create customizable web form fields. So you have structured information on the issues you need.

Issue forms are currently in beta.

You should have it in the same folder as the issue templates, but issue forms have a YAML extension. So, whenever a user needs to open a new issue, the user cannot change headings or completely remove everything, he won't be so evil.

Here we have a form that the user can fill out and select multiple values from, so the form will always be the same.

And this is totally amazing! Why? From here, it is so easy to automate anything you need.
So, let's say you need to run an script based on the inputs the user opening the form is providing. We can run a workflow from the issue created event, and the event body has the same format for every issue created; we can pick all the values provided and work with that.

Ok, but how? I created a repository called IssueChatbotTemplate so you can copy the workflow.

Let's analyze it here:

---
name: issue
run-name: 
on:
    issues:
    types: [opened]

jobs:
    issue:
        name: Get inputs from issue form
        runs-on: ubuntu-latest
        if: ${{ contains(github.event.issue.title , '[Test Issue]')}}
        timeout-minutes: 30
        steps:
            - name: Add Comment
            uses: actions/github-script@v6
            with:
                script: |
                github.rest.issues.createComment({
                    issue_number: context.issue.number,
                    owner: context.repo.owner,
                    repo: context.repo.repo,
                    body: 'Workflow running on: https://github.com/${{ github.repository }}/actions/runs/${{ github.run_id }}',
                });

            - name: Get inputs
            id: inputs
            shell: bash
            run: |
                add_output(){
                local k=$1
                local v=$2
                echo "$k=$v" >> $GITHUB_OUTPUT
                echo "✅ Output generated with key: $k, value: $v"
                }
                process_key(){
                    local process_key=$1
                    # Trim leading and trailing whitespace
                    process_key="$(echo -e "${process_key}" | sed -e 's/^[[:space:]]*//' -e 's/[[:space:]]*$//')"

                    # Substitute spaces with underscores
                    process_key="${process_key// /_}"

                    # Convert to lowercase
                    process_key="${process_key,,}"

                    echo "$process_key";
                }

                process_checkbox_line(){
                    local checkbox_line=$1
                    option=$(echo "$checkbox_line" | sed 's/- \[[^]]*\] //')

                    key="$(process_key "$option")"
                    value=$(echo "$checkbox_line" | grep -qE -- '- \[X\]' && echo "true" || echo "false")

                    add_output "$key" "$value"

                    escaped_lookfor=$(echo "$checkbox_line" | sed 's/[][()\.^$?*+{}|]/\\&/g')
                    nextcheckbox=$(echo "${{ github.event.issue.body }}" | awk "/$escaped_lookfor$/{getline; print}")

                    if [[ "$nextcheckbox" == "- ["* && -n "$nextcheckbox" && "$nextcheckbox" != "$checkbox_line" ]]; then
                        process_checkbox_line "$nextcheckbox"
                    fi
                }

                echo "${{ github.event.issue.body }}" | awk '/^###/ { print }' | while read line; do
                    echo "🔍 Processing line: $line"
                    value=$(echo "${{ github.event.issue.body }}" | awk "/^$line$/{getline;getline; print}")

                    if [[ "$value" == "- ["* ]]; then
                        # Processing a checkbox
                        echo "☑️ Detected a checkbox!"
                        process_checkbox_line "$value"
                    else
                        # Remove "###" from the beginning of the string
                        key=${line##"###"}

                        key="$(process_key "$key")"
                        add_output "$key" "$value"
                    fi
                done


            - uses: actions/github-script@v6
            name: Add label
            with:
                script: |
                github.rest.issues.addLabels({
                    issue_number: context.issue.number,
                    owner: context.repo.owner,
                    repo: context.repo.repo,
                    labels: ["environment/${{steps.inputs.outputs.dropdown_label}}"]
                });

        outputs:
            input: ${{ steps.inputs.outputs.input_label }}
            dropdown: ${{ steps.inputs.outputs.dropdown_label }}
            dropdown_multiple: ${{ steps.inputs.outputs.dropdown_multiple_label }}
            textarea: ${{ steps.inputs.outputs.textarea_label }}
            checkbox1: ${{ steps.inputs.outputs.checkbox_label_1 }}
            checkbox2: ${{ steps.inputs.outputs.checkbox_label_2 }}

    process:
        name: Process ${{needs.issue.outputs.dropdown}}
        runs-on: ubuntu-latest
        needs: issue
        steps:
        - name: Add Comment
            uses: actions/github-script@v6
            with:
            script: |
                const body = `### Processed outputs:

                **input_label**: ${{ needs.issue.outputs.input }}
                **dropdown_label**: ${{ needs.issue.outputs.dropdown }}
                **dropdown_multiple_label**: ${{ needs.issue.outputs.dropdown_multiple }}
                **textarea_label**: ${{ needs.issue.outputs.textarea }}
                **checkbox_label_1**: ${{ needs.issue.outputs.checkbox1 }}
                **checkbox_label_2**: ${{ needs.issue.outputs.checkbox2 }}
                `;

                github.rest.issues.createComment({
                issue_number: context.issue.number,
                owner: context.repo.owner,
                repo: context.repo.repo,
                body: body
                });

Here I'm running the workflow whenever a new issue is created; if the issue has the title [Test Issue] I run the first job (Yes, the user can remove the [Test Issue] title, you can manage it with labels, for example. And yes, the user can remove the label also.).
Then, in the first step, I'm adding a comment to the same issue, so the user knows the workflow is running.
The next step is where magic happens: grabbing all inputs and adding them as GitHub outputs with the label formatted and the value.
Also, I'm adding some labels to the issue based on the input provided, so for example, you can know which environment is faster. Adding outputs for this operation is the final step in the job. That's where the work should end.

And finally, in a new job, adding a comment with the id and the value known in the same issue (this is something you probably won't need, just for debugging purposes).

So, as a result, we will have something like that:

And this is just a start point, from here you can grow it by calling another job to deploy the application to some environment, running an automation to grant some permissions for some user (For security changes, you will need to check the user creating the issue, of course), or any other idea you want to automate.
Also, you can improve it. So let's say you have like 3-4 issue forms in the new template, so you have 3-4 workflows for these issue forms.
Whenever you open a new issue, all workflows will be executed, but not the jobs. So you can improve it by changing the event of these workflows to workflow_call and having 1 main workflow that adds the first comment in the issue, and then with a strategy calling the workflow you want to execute based on title, labels, etc...

Thanks for your time reading this! I highly appreciate any feedback on any of my social media accounts or here.

Autoscale for Cosmos DB

David Rodríguez Donoso — Wed, 31 Jul 2019 11:05:14 +0000

If you have ever used Cosmos DB in your project, maybe you realized that one of the main concerns of using Cosmos DB is that it doesn't have autoscale. Microsoft started to work on this two years ago.

One problem is when you are having lots of requests in a range of hours or maybe because something triggers that lots of users use the application at the same time.

How to do

First of all we will need the method to modify the offer. For this I added some methods to get the offers and scale it:

private async Task ScaleAsync(string database, string collection, int scaleBatch, int minThroughput, int maxThroughput)
{
    var offer = await GetOfferAsync(database, collection);
    await ScaleAsync(offer, scaleBatch, minThroughput, maxThroughput);
}

private async Task ScaleAsync(OfferV2 offer, int scaleBatch, int minThroughput, int maxThroughput)
{
    var currentThroughput = offer.Content.OfferThroughput;

    var newThroughput = currentThroughput + scaleBatch;
    if (newThroughput < minThroughput)
    {
        newThroughput = minThroughput;
    }

    if (newThroughput > maxThroughput)
    {
        newThroughput = maxThroughput;
    }
    var updatedOffer = new OfferV2(offer, newThroughput);
    await documentClient.ReplaceOfferAsync(updatedOffer);
}

private async Task<OfferV2> GetOfferAsync(string database, string collection)
{
    return await GetCollectionOfferAsync(database, collection) ?? await GetDatabaseOfferAsync(database);
}

private async Task<OfferV2> GetCollectionOfferAsync(string database, string collection)
{
    var collectionUri = UriFactory.CreateDocumentCollectionUri(databaseName, collectionName);
    var collection = (await _documentClient.ReadDocumentCollectionAsync(collectionUri)).Resource;

    return await GetOfferAsync(collection.SelfLink);
}

private async Task<OfferV2> GetDatabaseOfferAsync(string database)
{
    var databaseUri = UriFactory.CreateDatabaseUri(databaseName);
    var database = (await _documentClient.ReadDatabaseAsync(databaseUri)).Resource;

    return await GetOfferAsync(database.SelfLink);
}

private async Task<OfferV2> GetOfferAsync(string selfLink)
{
    return (await _documentClient.CreateOfferQuery()
                .Where(o => o.ResourceLink == selfLink)
                .AsDocumentQuery()
                .ExecuteNextAsync<OfferV2>()).FirstOrDefault();
}

Scale up

When there is a request that is exceeding the provisioned throughput that you have for the database or collection, Cosmos is throwing a DocumentClientException with the status code 429.
So, what we should do is to catch this exception and scale up this database or collection and retry the execution:

public async Task<T> ExecuteAsync<T>(Func<IDocumentClient, Task<T>> func, string database, string collection, int scaleUpBatch, int minThroughput, int maxThroughput, int retries = 0)
{
    try
    {
        return await func(_documentClient);
    }
    catch (DocumentClientException exception) when ((int)exception.StatusCode == 429)
    {
        retries++;
        await ScaleAsync(database, collection, scaleUpBatch, minThroughput, maxThroughput);
        if (retries < _config.MaxRetries) return await ExecuteAsync(func, database, collection, scaleUpBatch, minThroughput, maxThroughput, retries);
        throw;
    }
}

Scale down

To scale down we should use a cronjob, I was using Azure Functions to scale down all the collections.

public async Task ScaleDownAll(int scaleDownBatch, int minThroughput, int maxThroughput)
{
    var scaleBatch = Math.Abs(scaleDownBatch) * -1;
    var databases = _documentClient.CreateDatabaseQuery().AsEnumerable().ToList();

    foreach (var database in databases)
    {
        var offer = await GetOfferAsync(database.SelfLink);

        if (offer != null)
        {
            await ScaleAsync(offer, scaleBatch, minThroughput, maxThroughput);
        }
        else
        {
            var collections = _documentClient.CreateDocumentCollectionQuery(database.SelfLink).ToList();
            foreach (var collection in collections)
            {
                offer = await GetOfferAsync(collection.SelfLink);
                await ScaleAsync(offer, scaleBatch, minThroughput, maxThroughput);
            }
        }
    }
}

If you want to see more about this, I started a repo in Github:

https://github.com/DrDonoso/Cosmos-Autoscale