DEV Community: Arthur Diniz

Live Preview

Arthur Diniz — Fri, 26 Nov 2021 02:47:17 +0000

My Workflow

This project aims to provide temporary validation deployments that make the process of reviewing a PR easier by providing an environment in which the reviewer can verify if the changes are good-to-go from a user perspective. Its main advantage is to remove the requirement for pulling the branch code, building and running it locally.

Whenever the action is triggered, it will pull the application's Docker image (or images, if the project requires a docker-compose with multiple services), run it and magically expose it to the reviewers, posting the link to the application in a comment. The magic can be handled either by Ngrok, a freemium tunneling service, or an open-source, self-hosted, FRP (Fast Reverse Proxy) server that works as a reverse proxy and tunneling platform.

As a demo we created a simple Dockerfile with nginx and a static html page: https://github.com/overhead-actions/live-preview-demos/pull/1

overhead-actions / live-preview-demos

Live Preview Demo

This is a demo for the Live Preview project:

Ngrok Version: github.com/overhead-actions/live-preview-ngrok
FRP (Fast Reverse Proxy) Version: github.com/overhead-actions/live-preview-frp

The Live Preview project aims to provide temporary validation deployments that make the process of reviewing a PR easier, by providing an environment in which the reviewer can verify if the changes are good-to-go from a user perspective. Its main advantage is to remove the requirement for pulling the branch code, building and running it locally.

Whenever the action is triggered, it will pull the application's Docker image (or images, if the project requires a docker-compose with multiple services), run it and magically expose it to the reviewers, posting the link to the application in a Pull Request comment. The magic can be handled either by Ngrok, a freemium tunneling service, or an open-source, self-hosted, FRP (Fast Reverse Proxy) server that works as a reverse proxy and tunneling…

View on GitHub

Submission Category:

Maintainer Must-Haves

Yaml Files

FRP (Fast Reverse Proxy)

name: Live Preview FRP

on: pull_request

jobs:
  default:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v2

      - name: Start services
        run: docker-compose up -d

      - name: Start tunnel
        uses: overhead-actions/live-preview-frp@main
        with:
          domain: ${{ github.head_ref }}.arthurbdiniz.com

      - name: Comment PR
        uses: unsplash/comment-on-pr@v1.3.0
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
        with:
          msg: 'Here is your live preview URL 🚀: http://${{ github.head_ref }}.arthurbdiniz.com'
          check_for_duplicate_msg: false

      - name: Wait
        run: sleep 300

Ngrok

name: Live Preview NGROK

on: pull_request

jobs:
  default:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v2

      - name: Start services
        run: docker-compose up -d

      - name: Start tunnel
        uses: overhead-actions/live-preview@main
        with:
          protocol: http
          port: 4000
          ngrok_auth_token: ${{ secrets.NGROK_AUTH_TOKEN }}

      - name: Get URL
        id: vars
        run: echo "::set-output name=url::$(curl -s localhost:4040/api/tunnels | jq -r .tunnels[0].public_url)"

      - name: Comment PR
        uses: unsplash/comment-on-pr@v1.3.0
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
        with:
          msg: 'Here is your live preview URL 🚀: ${{ steps.vars.outputs.url }}'
          check_for_duplicate_msg: false

      - name: Wait
        run: sleep 300

Collaborators

Dropdown for GitHub workflows input parameters

Arthur Diniz — Mon, 25 Oct 2021 22:09:59 +0000

Dropdown for GitHub workflows input parameters

Sometimes when we look at CI/CD tools embedded within git-based software repository manager like GitHub, GitLab or Bitbucket, we ran into a lack of some features.

This time me and my DevOps/SRE team were facing a pain of not being able to have the option to create drop-downs within GitHub workflows using input parameters. Although this functionality is already available on other platforms such as Bitbucket, the specific client we were working on stored the code inside GitHub.

At first I thought that someone has already solved this problem somehow, but doing an extensive search on the internet I found several angry GitHub users opening requests within the Support Community and even in the stack overflow.

So I decided to create a solution for this, always thinking about simplicity and in a way that makes it easy to get this missing functionality. I started by creating an input array pattern using commas and using a tag (the selector) e.g brackets as the default value marker. Here is an example of what an input string would look like:

name: gh-action-dropdown-list-input
on:
  workflow_dispatch:
    inputs:
      environment:
        description: 'Environment'
        required: true
        default: 'dev,staging,[uat],prod'

Now the final question that would turn out to be the most complicated to deal with. How can I change the GitHub Actions interface to replace the input pattern we created earlier to a dropdown?

The simplest answer I thought was to create a chrome and firefox extension that would do all this logic behind the scenes and replace the HTML input element with the selected tag containing the array values and leaving the tag value (selector) always as the default.

All code was developed in pure JavaScript, open-source licensed under Apache 2.0 and available at https://github.com/arthurbdiniz/gh-action-dropdown-list-input.

Install extension

Once installed, the extension is ready to use and the final result we see is the Actions interface with drop-downs. :)

Configuring selectors

Go to the top right corner of the browser you are using and click on the extension logo. A screen will popup with tag options. Choose the right tags for you and save it.

This action might require reloading the GitHub workflow tab.

Have fun using drop-downs inside GitHub. If you liked this project please share this post and if possible "star" within the repository.

Also feel free to connect with me on LinkedIn: https://www.linkedin.com/in/arthurbdiniz

References

Upgrade Elasticsearch from 2.3 to 7.4

Arthur Diniz — Thu, 21 Oct 2021 06:24:20 +0000

Sometimes it is common just to deploy a Elasticsearch domain at AWS and forget to keep it up-to-date with the latest version. After a while, you realize that it's five major releases behind the stable version and there's no straight path to upgrade.

The idea of this post is to show the problems that you can face during this process of upgrading and also give you a more simplified way than following the AWS documentation.

Pre-requirements

ES domain v2.3
ES domain v5.1 (Optional)
IAM account with S3 and ES permissions

From 2.3 to 5.1

According to the AWS documentation, the first upgrade needs to be done manually using the ES snapshot API. So to successfully upgrade, you will need to create an S3 bucket and a set of permissions that allow your ES domains to dump data into the bucket.

To save your time, we have created a python script that communicates with the AWS API and setup everything for us. Focused on avoiding problems with your root domain, this script will create a brand new domain and execute all the steps in there.

All the code can be found at: es-auto-upgrade

Script steps

Create an S3 bucket to store snapshot from 2.3 domain
Create new Elasticsearch domain (Optional)
Create IAMpermissions (Policy, Role, and attachments)
Register snapshot in both domains
Take snapshot from 2.3 and save it to S3 bucket
Restore snapshot from S3 into the new domain

Teardown

Delete S3 bucket
Delete IAMpermissions (Policy, Role, and detachments)

Dependencies

Docker

Variables

# Elasticsearch
export OLD_DOMAIN_NAME=test
export AWS_REGION=ap-southeast-2
export NEW_DOMAIN_NAME=test-new
export CREATE_NEW_DOMAIN=True # Optional
export NEW_INSTANCE_TYPE=m5.xlarge.elasticsearch # Optional
# S3
export BUCKET_NAME=es-automated-update # Optional

To run all of these steps at once just run:

# Download the required variables, you should edit this file.
wget https://raw.githubusercontent.com/DNXLabs/es-auto-upgrade/master/var.env
docker run -it --env-file vars.env dnxsolutions/es-auto-upgrade upgrade.py

When the script finishes running, you should have your brand new ES domain at version 5.1 with all your data from 2.3.

From 5.1 to 7.4

From now on Amazon ES offers in-place Elasticsearch upgrades for domains that run versions 5.1 and later and we can proceed using the In-place along with the Elasticsearch reindex API to get to version 7.4.

Currently, Amazon ES supports the following upgrade paths.

5.1 to 5.6 and 5.6 to 6.8

Important:
Indices created in version 6.x no longer support multiple mapping types. Indices created in version 5.x still support multiple mapping types when restored into a 6.x cluster. Check that your client code creates only a single mapping type per index.
To minimize downtime during the upgrade from Elasticsearch 5.6 to 6.x, Amazon ES reindexes the .kibana index to .kibana-6, deletes .kibana, creates an alias named .kibana, and maps the new index to the new alias.

6.8 to 7.4

Important:
Elasticsearch 7.0 includes numerous breaking changes. Before initiating an in-place upgrade, we recommend taking a manual snapshot of the 6.8 domain, restoring it on a test 7.x domain, and using that test domain to identify potential upgrade issues.
Like Elasticsearch 6.x, indices can only contain one mapping type, but that type must now be named _doc. As a result, certain APIs no longer require a mapping type in the request body (such as the _bulk API).
For new indices, self-hosted Elasticsearch 7.x has a default shard count of one. Amazon ES 7.x domains retain the previous default of five.

This next steps will repeat until you get to the latest ES version.

1. Reindex

docker run -it --env-file vars.env dnxsolutions/es-auto-upgrade reindex.py

2. Rollout Upgrade

Go to your domain actions and select the Upgrade Domain button.

Then select the Upgrade checkbox and Submit.

AWS now will take care of the rest taking snapshots and upgrading to the next ES version. You can check the progress at the Upgrade History tab.

Once this step is finished, repeat the reindex and upgrade until you get to 7.4.

Troubleshooting

Your domain sometimes might be ineligible for an upgrade or fail to upgrade for a wide variety of reasons. So AWS has provided one table showing the most common issues.

https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/es-version-migration.html#upgrade-failures

Mapping Issues

When you get to 7.4, you might notice some changes with the mapping. Since 2.3, a lot of new features were implemented, and this causes a lot of changes with mapping.
So what could happen is that you have a variable title with called id and it has changed to string.
What you can do it take the new mapping consulting the ES API at the endpoint.

https://$ES_DOMAIN_URL/<indice>/_mappin

Then compare with the original mapping at version 2.3 and adapt to your needs.

file: mapping.json
{
    "mappings":{
        "properties":{
            "style": {
                "type": "text",
                "analyzer": "keyword"
            }
        }
    }
}

# Apply the new mapping
for index in <indice>; do
    curl -HContent-Type:application/json -XPUT "$ES_DOMAIN_URL/$index-new" -d @mapping.json
done

String -> Text

{
    "mappings":{
        "properties":{
            "id": {
                "type": "text"
            }
        }
    }
}

Float -> Long

{
    "mappings":{
        "properties":{
            "price":{
                "type":"long"
            }
        }
    }
}

Completion type

If you use completion type with payload and output, you may notice that since v5.1, ES does not support these two fields anymore. So to find a workaround for this, the following issue did this very well.

Now, instead of returning the payload, the whole hit is returned from Elasticsearch, so you can use any regular property of the document. In order to save bandwidth and memory, it makes sense to return only properties which you care about, by using the "_source filtering" feature (see _source_include and _source_exclude features.

https://github.com/elastic/elasticsearch-rails/issues/690#issuecomment-290706168

So, for example, this is one way of using context, payload, and output:

"brand_suggestion":{
   "properties":{
      "context":{
         "properties":{
            "visible_context":{
               "type":"boolean"
            }
         }
      },
      "input":{
         "type":"completion",
         "analyzer":"simple",
         "preserve_separators":true,
         "preserve_position_increments":true,
         "max_input_length":50,
         "contexts":[
            {
               "name":"visible_context",
               "type":"CATEGORY",
               "path":"is_visible"
            }
         ]
      },
      "output":{
         "type":"text",
         "fields":{
            "keyword":{
               "type":"keyword",
               "ignore_above":256
            }
         }
      },
      "payload":{
         "properties":{
            "id":{
               "type":"long"
            },
            "urlKey":{
               "type":"text",
               "fields":{
                  "keyword":{
                     "type":"keyword",
                     "ignore_above":256
                  }
               }
            }
         }
      }
   }
}

Conclusion

We hope we have helped with some insights into your Amazon Elasticsearch Service migration.

It is not an easy job, and if you have a big distributed cluster, you may want to use other tools to handle this amount of workload.