DEV Community: Laysa Uchoa

🔐 Google Cloud Authentication with Workload Identity Federation for GitHub Actions

Laysa Uchoa — Thu, 21 Dec 2023 21:04:57 +0000

Introduction

In the past, when you wanted to allow an application like GitHub to access your Google Cloud resources, you had to follow a few steps. First, you would create a separate IAM service account for each application. Then, you would download a long-lasting JSON service account key and store it in your GitHub repository's secrets. These service account keys could then be used to make GCP API calls from your application.

Managing these service account keys involved secure storage, regular rotation, and continuous monitoring to protect your Google resources from potential malicious actors. If a malicious actor gained access to your keys, they could maintain prolonged access to your Google Cloud resources. This is why this approach is not recommended.

So, what's the solution? The answer is Workload Identity Federation (WIF).

Through the Workload Identity Federation (WIF), you can grant external identities IAM roles, by allowing them to impersonate service accounts that have these roles.

This approach is superior for the following reasons:

You no longer need to manage key rotation or store service account keys.
The credentials have a short default lifespan and are configurable. This approach simplifies the management and security aspects associated with service account keys.

Steps to Set Up Workload Identity Federation

Step 1: Create a Workload Identity Pool

According to the documentation, a Workload Identity Pool (WIP) is an entity that lets you manage external identities. In general, GCP recommends creating a new pool for each non-Google Cloud environment that needs to access Google Cloud resources, such as development, staging, or production environments.

In this step, we are defining the locals to be used, and we are using a unique name for the Workload Identity Pool (WIP).

locals {
  project_id = "PROJECT_ID"
  service_account = "SERVICE_ACCOUNT"
  organization     = "YOUR_ORGANISATION_HERE"
  repository       = "YOUR_REPOSITORY_HERE"
}


resource "google_iam_workload_identity_pool" "github_pool" {
  project                   = local.project_id
  workload_identity_pool_id = "github-pool-oidc"
  display_name              = "GitHub pool"
  description               = "Identity pool for GitHub deployments"
}

Step 2: Configure the Workload Identity Pool Provider

In this step, we configure the Workload Identity Pool provider. Your provider should have a unique name. Then, we can set the "Attribute Mapping" to map the GitHub Actions attributes to GCP identity attributes.

Here, we can also define an "Attribute Condition" which is recommended to be more secure. This condition specifies when the identity provider should be used based on GitHub repository attributes, for example, repository name, repository owner, actor, and others.

We also set up the OIDC settings by providing the correct issuer URI.

resource "google_iam_workload_identity_pool_provider" "github" {
  project                            = local.project_id
  workload_identity_pool_id          = google_iam_workload_identity_pool.github_pool.workload_identity_pool_id
  workload_identity_pool_provider_id = "github-provider"

  attribute_mapping = {
    "attribute.aud"              = "assertion.aud"
    "google.subject"             = "assertion.sub"
    "attribute.sub"              = "assertion.sub"
    "attribute.actor"            = "assertion.actor"
    "attribute.repository"       = "assertion.repository"
    "attribute.repository_owner" = "assertion.repository_owner"
    "attribute.ref"              = "assertion.ref"
  }

  # If you want to restrict to organisation, use:
  # "assertion.repository_owner==\"${local.organization}\""

  # For more than one repository, use:
  # "assertion.repository==\"ORG/repository-1\" || assertion.repository==\"ORG/repository-2\""

  attribute_condition = "assertion.repository==\"${local.organization}/${local.repository}\""


  oidc {
    allowed_audiences = []
    issuer_uri        = "https://token.actions.githubusercontent.com"
  }
}

Step 3: Configure IAM Roles

In this step, we configure the GCP IAM role for our service account. This can be done by setting the roles/iam.workloadIdentityUser to the service account we want to impersonate.

What this role does is to allow the associated service account to assume identities from a trusted identity provider to access Google Cloud resources which this service account has permission for.

Step 5: Full Terraform code

To sum up, this is the full terraform code we needed for the setup.

# wif.tf

locals {
  project_id = "PROJECT_ID"
  service_account = "SERVICE_ACCOUNT"
  project_name    = "YOUR_PROJECT_NAME"
  organization     = "YOUR_ORGANISATION_HERE"
  repository       = "YOUR_REPOSITORY_HERE"
}

resource "google_iam_workload_identity_pool" "github_pool" {
  project                   = local.project_id
  workload_identity_pool_id = "github-pool-oidc"
  display_name              = "GitHub pool"
  description               = "Identity pool for GitHub deployments"
}

resource "google_iam_workload_identity_pool_provider" "github" {
  project                            = local.project_id
  workload_identity_pool_id          = google_iam_workload_identity_pool.github_pool.workload_identity_pool_id
  workload_identity_pool_provider_id = "github-provider"

  attribute_mapping = {
    "attribute.aud"              = "assertion.aud"
    "google.subject"             = "assertion.sub"
    "attribute.sub"              = "assertion.sub"
    "attribute.actor"            = "assertion.actor"
    "attribute.repository"       = "assertion.repository"
    "attribute.repository_owner" = "assertion.repository_owner"
    "attribute.ref"              = "assertion.ref"
  }

  # If you want to restrict the organization, use the:
  # "assertion.repository_owner==\"${local.organization}\""

  # For more than one repository, use:
  # "assertion.repository==\"ORG/repository-1\" || assertion.repository==\"ORG/repository-2\""

  # For one repository:
  attribute_condition = "assertion.repository==\"${local.organization}/${local.repository}\""

  oidc {
    allowed_audiences = []
    issuer_uri        = "https://token.actions.githubusercontent.com"
  }

}

resource "google_service_account_iam_member" "workload_identity_user" {
  service_account_id = 'projects/${local.project_name}/serviceAccounts/${local.service_account}' #Replace with your service account
  role               = "roles/iam.workloadIdentityUser"
  member             = "principalSet://iam.googleapis.com/${google_iam_workload_identity_pool.github_pool.name}/attribute.repository_owner/${local.organization}"
}

Step 6: Setup GitHub Actions Workflow

You need to save as secrets the provider_name as this is sensitive information. It has the following format:

WORKLOAD_IDENTITY_PROVIDER = projects/PROJECT_NUMBER/locations/global/workloadIdentityPools/github-pool-oidc/providers/github-provider

Below is an example of a workflow using GCP WIF OpenID Connect (OIDC) in GitHub Actions.

name: Example GCP WIF with GitHub Actions

on:
  push:
    branches:
      - setup-wif-oidc

jobs:
  job_id:
    runs-on: ubuntu-latest
    permissions:
      contents: 'read'
      id-token: 'write'

    steps:
      # actions/checkout MUST come before auth
      - uses: 'actions/checkout@v3'

      - id: "auth"
        name: "Authenticate to Google Cloud"
        uses: "google-github-actions/auth@v2"
        with:
          token_format: "access_token"
          workload_identity_provider: ${{ secrets.WORKLOAD_IDENTITY_PROVIDER }}
          service_account: YOUR_SERVICE_ACCOUNT
          export_environment_variables: true
          audience: ${{ secrets.GCP_POOL_AUDIENCE }}
          create_credentials_file: true
          access_token_lifetime: 500
      # ... further steps are automatically authenticated

      - name: "Set up Cloud SDK"
        uses: "google-github-actions/setup-gcloud@v1"
        with:
          version: ">= 390.0.0"

      - name: Check currently authenticated user
        run: gcloud auth list

      # interact with google cloud
      - name: Run gcloud
        run: gcloud storage buckets list

The workflow authenticates with Google Cloud, sets up necessary tools, and performs a listing of storage buckets within the Google Cloud environment.

Step 7: GitHub Action in Action

Find here how the output of the run looks like:

More Ressources

For more information on best practices for using WIF on Google Cloud: https://cloud.google.com/iam/docs/best-practices-for-using-workload-identity-federation

-
👋 Hello, I'm Laysa, a developer and cloud engineer, a public speaker, and a knowledge sharer as I go along.

♻️ Share this article, if you like the content.

💜 Find me everywhere | X | LinkedIn

Get OpenSearch mapping with Python

Laysa Uchoa — Sat, 16 Dec 2023 15:58:11 +0000

When no data structure is specified when loading the data, OpenSearch uses dynamic mapping to automatically detect the fields. To check the mapping definition of your data, OpenSearch client provides a function called get_mapping as shown:

    import pprint

    INDEX_NAME = 'epicurious-recipes'
    mapping_data = os_client.indices.get_mapping(INDEX_NAME)

    # Find index doc_type
    doc_type = list(mapping_data[INDEX_NAME]["mappings"].keys())[0]

    schema = mapping_data[INDEX_NAME]["mappings"][doc_type]
    fields =  list(schema.keys())
    pprint(fields)
    pprint(schema)

You should be able to see the fields's output:


    ['calories',
    'categories',
    'date',
    'desc',
    'directions',
    'fat',
    'ingredients',
    'protein',
    'rating',
    'sodium',
    'title']

And the mapping with the fields and their respective types.


        {'calories': {'type': 'float'},
         'categories': {'fields': {'keyword': {'ignore_above': 256, 'type': 'keyword'}},
                        'type': 'text'},
         'date': {'type': 'date'},
         'desc': {'fields': {'keyword': {'ignore_above': 256, 'type': 'keyword'}},
                  'type': 'text'},
         'directions': {'fields': {'keyword': {'ignore_above': 256, 'type': 'keyword'}},
                        'type': 'text'},
         'fat': {'type': 'float'},
         'ingredients': {'fields': {'keyword': {'ignore_above': 256,
                                                'type': 'keyword'}},
                         'type': 'text'},
         'protein': {'type': 'float'},
         'rating': {'type': 'float'},
         'sodium': {'type': 'float'},
         'title': {'fields': {'keyword': {'ignore_above': 256, 'type': 'keyword'}},
                   'type': 'text'}}

Read more about OpenSearch mapping in the official OpenSearch documentation.

A Decade of Docker! 🐋

Laysa Uchoa — Fri, 17 Nov 2023 19:09:24 +0000

Can you believe Docker first showed up in a lightning talk that got everyone talking? Today, I'd like to talk about the history of Docker 🐋

The Birth of Docker

It all began over a decade ago. Docker, introduced in a lightning talk at PyCon US in 2013, was the brainchild of Solomon Hykes, an entrepreneur and programmer. Hykes, deeply immersed in the world of software development, sought to address a common challenge faced by developers: complexity and inconsistency in deploying applications across various environments.

So, what was the problem? It was the notorious "it worked on my machine issue." Developers often encountered scenarios where their code worked perfectly on their development environment but failed to function correctly when deployed elsewhere. This inconsistency was a significant pain point in software development.

Hykes envisioned a platform that could package applications and their dependencies into lightweight, portable containers. These containers, he believed, could be effortlessly deployed and executed consistently on any system, from a developer's laptop to a production server.

The Birth of "Docker"

The name "Docker" was chosen to evoke the idea of shipping containers, which are standardized, easily transportable, and capable of holding a variety of goods 📦. Docker containers, in a similar vein, could hold software applications and their dependencies, making them transportable across different computing environments.

With Docker, developers could now create, share, and run applications in isolated, reproducible containers, eliminating the notorious "it works on my machine" problem.

Docker's Journey Unfolds

Docker's journey unfolds as follows:

2013: Solomon Hykes introduces Docker in a lightning talk at PyCon US, marking its inception.
2013-2014: Docker gains early adopters and community support as developers recognize its potential.
2015: Docker, Inc. is formed to provide commercial support and services around Docker.
Late 2010s: Docker becomes a fundamental tool in the DevOps toolkit, used by organizations worldwide.
2020s: Docker continues to evolve and adapt to the changing needs of the software development industry.

A Developer's Dream

Docker revolutionized the way developers worked. It allowed developers to concentrate on writing code while abstracting away many of the infrastructure complexities. No longer did they have to worry about the intricate details of setting up development and production environments. Docker containers provide a consistent and efficient way to manage applications and their dependencies.

First World Appearance of Docker

Here you can find the first world appearance of Docker 📺 ⭐ Docker Lightning Talk at PyCon US 2013.

Happy learning! 👋

I'm a Cloud Engineer at Nordcloud.

Automating Terraform Deployments with GitHub Actions: A Step-by-Step Guide

Laysa Uchoa — Thu, 02 Nov 2023 21:01:43 +0000

I'm a Cloud Engineer at Nordcloud.

Introduction

GitHub Actions is a potent automation tool that elevates your software development processes by responding to events and executing predefined workflows.

In this article, you will learn how to use GitHub Actions to deploy your Terraform code to production. 🔄

Scenario

You've been working on your infrastructure as code (IaC) using Terraform. Now, you're preparing to implement your changes in a production environment. You need these deployments to be error-free, follow best practices, and have appropriate approval procedures. GitHub Actions can be effective for this by using a manual step before deploying to production.

Setup

Setting up your Terraform deployment workflow with GitHub Actions involves several important steps. In this section, we'll walk you through each step in detail.

1) Configure GitHub Actions trigger and permissions

The first step in automating your Terraform deployments is to configure the GitHub Actions workflow trigger and permissions. This configuration determines when and how the workflow will be initiated.

Here's the YAML code to configure GitHub Actions:



#.github/workflows/deploy-to-prod.yaml
name: Terraform PROD Deployment

on:
  push:
    branches:
      - main
  workflow_dispatch:
    branches:
      - main

permissions:
  contents: read
  issues: write

This configuration accomplishes two key things:

Trigger Events: It specifies that the workflow should be triggered when changes are pushed to the "main" branch and when manually initiated using the "workflow_dispatch" event.
Permissions: It defines the necessary permissions for the workflow, ensuring that it can read repository contents and write issues.

2) Add initial Setup

Let's examine the workflow steps:

Checkout Git Repository: This step involves checking out your Git repository.
Authenticate with your cloud provider: This is crucial if you need to deploy resources, like a Terraform deployment, on your cloud provider. In this example, we are using GCP (Google Cloud Provider) for the deployment.



jobs:
  gcp-deploy:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout Git Repository
        uses: actions/checkout@v3.0.0
        with:
          fetch-depth: 0

      - id: "auth"
        name: "Authenticate to Google Cloud"
        uses: "google-github-actions/auth@v1"
        with:
          credentials_json: "${{ secrets.GOOGLE_CREDENTIALS }}"
          create_credentials_file: true
          export_environment_variables: true

3) Check the Terraform code

Ensure Terraform code works in your local setup: Do this before using it directly in your workflow to avoid errors related to your Terraform code itself rather than GitHub Actions errors.

You can try your terraform commands such as init, plan and apply:



terraform init
terraform plan
terraform apply # Review your plan before applying

4) Add Terraform steps

In this step, we'll use terraform init to get your Terraform environment ready by setting up what it needs. Then, we'll run 'terraform plan' to see a preview of the changes that will be made.



- name: Setup Terraform
  uses: hashicorp/setup-terraform@v1
  with:
      terraform_version: 1.5.7
- name: Terraform init and validate
  run: |
      terraform init

- name: Terraform plan
    run: |
        terraform plan

5) Create Issue for approval

You can use GitHub Actions, for example, to deploy your Terraform code, but before applying the changes, you need a mechanism to review your Terraform code. This helps ensure that your infrastructure changes are safe and minimizes the risks of deployment in production

For the manual approval in the workflow, we will use the trstringer/manual-approval .

This GitHub Actions workflow enables you to create an approval process within your automation pipeline. It's particularly useful when you want to add a layer of human review and approval, especially for actions such as deploying to production. 👀



- uses: trstringer/manual-approval@v1
  with:
    secret: ${{ github.TOKEN }}
    approvers: laysauchoa
    minimum-approvals: 1
    issue-title: "Deploying v1.0.0 to prod"
    issue-body: "Review the terraform plan, then approve or deny the deployment of v1.0.0 to prod."
    exclude-workflow-initiator-as-approver: false

Once the Terraform plan step is completed, a corresponding issue is automatically generated for your review. Below, you can see an example of what this issue might look like:

In this issue, you have the opportunity to thoroughly examine the proposed changes in your Terraform deployment. If you agree with the plan and are ready to proceed with the deployment, you can simply approve the issue.

Upon approval, the workflow will automatically advance to the next step, applying your Terraform changes.

Deploying changes to production can be risky. Using a manual approval process allows us to review and confirm the deployment plan before proceeding, especially when deploying to production.

Conclusion

That's it! You've successfully built a CI/CD pipeline using Terraform and GitHub Actions. Just think about how much time and hassle this can save you as your website or system grows. Terraform and GitHub Actions make things easier and faster.

See you next time! 👋



#.github/workflows/deploy-to-prod.yaml
name: Terraform PROD Deployment

on:
    push:
      branches:
        - develop
    workflow_dispatch:
      branches:
        - develop

permissions:
    contents: read
    issues: write

jobs:
  gcp-deploy:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout Git Repository
        uses: actions/checkout@v3.0.0
        with:
          fetch-depth: 0

      - id: "auth"
        name: "Authenticate to Google Cloud"
        uses: "google-github-actions/auth@v1"
        with:
          credentials_json: "${{ secrets.GOOGLE_CREDENTIALS }}"
          create_credentials_file: true
          export_environment_variables: true

      - name: Setup Terraform
        uses: hashicorp/setup-terraform@v1
        with:
            terraform_version: 1.5.7

      - name: Terraform init and validate
        run: |
          terraform init

      - name: Terraform plan
        run: |
          terraform plan

      - uses: trstringer/manual-approval@v1
        with:
          secret: ${{ github.TOKEN }}
          approvers: laysauchoa
          minimum-approvals: 1
          issue-title: "Deploying v1.0.0 to prod"
          issue-body: "Review the terraform plan, then approve or deny the deployment of v1.0.0 to prod."
          exclude-workflow-initiator-as-approver: false

      - name: Terraform apply
        run: |
          terraform apply

Automate GitHub Pull Requests with GitHub Actions

Laysa Uchoa — Sat, 22 Oct 2022 20:37:37 +0000

Whenever you catch yourself in a repetitive task, try this: step back and think about ways you might automate it. One tool CI/CD folks love to use for automation is GitHub Actions.

GitHub Actions are widely used to run tests and perform checks on our code. But what happens when your automation changes the code, and you want to commit those changes? If you are thinking about running the script manually and opening a pull request, this is an option. But here's an even better option: automate the process by using GitHub Actions to open pull requests for you. This article shows you how, step by step.

Scenario

At Aiven, we've got an interesting problem: we keep expanding our list of cloud providers. This is nice because it gives you the flexibility to choose whatever provider works for you. For us, maintainers of dev portal documentation, it's a challenge to ensure that the most up-to-date information about our cloud providers is available for our readers.

You can find a list of Aiven's cloud providers on [Developer.aiven.io], which is also the portal where we host the developer documentation. To quickly retrieve a list of currently-available cloud providers, my colleague wrote a Python script that pulls our cloud listing from the Aiven API and generates documentation, which is really cool!

Having the script was great. Now, by running the script, we could quickly generate fresh documentation out of the Aiven API data. This didn't automate the process entirely, though. Getting the changes merged still required a number of manual actions on a regular basis: run the script, check if there are new changes to commit, create pull requests, etc.

This is a good example where you can configure a GitHub Actions workflow to run the script periodically and open a pull request when changes are detected. So if you have a similar challenge, below you'll find how to add an automation flow step by step.

Configure GitHub Actions workflow

Time for some action - or better, GitHub Actions! You can follow these steps to configure your GitHub Actions workflow:

1) Check the script

Let's ensure that the script works correctly before using it for automation. So, here you can run your script a couple of times and check that it is working as it is supposed to.

To make the process easier, you can add the commands to run the script in a Makefile. Makefiles are a helpful way to organize code compilation, and it helps to run long commands in a shorter version using the make command. Following our example, here is what we add to the Makefile:

```
SOURCEDIR     = .
# (Re)Generate cloud listing
cloud-list:
    python "$(SOURCEDIR)/scripts/aiven/clouds.py" "includes/clouds-list.rst"
```

Now we can simply run the script with:

```
make cloud-list
```

This step ensures that the script you use to change the code produces the changes you want to commit in a pull request. Mistakes happen. If your script isn't working as expected, you may need to debug your code before proceeding. If the script is working, then you can go ahead and create the GitHub workflow.

2) Create a GitHub workflow

In this step, you need to add a YAML file to define the workflow configuration. First, make sure you have created a .github/workflows/ folder; if not, you can go ahead and create the directory. Then, create a YAML file to define the workflow configuration, for example, .github/workflows/cloud-list.yaml. In this file, you can define the name of the workflow. In our example, we start by giving a name to our workflow:

```
name: Cloud - Create PR to update available list
```

Pick a name for a file and for the workflow that is related to the changes you are bringing to your code. This can help your code and to identify your workflows. Check out this quick guide from GitHub in how to create your first workflow.

3) Choose a trigger

GitHub Actions enable you to trigger your workflow to run based on a certain event or events. In the example, the job needs to run periodically. For that, configure it to run on schedule according to a cron job.

The cron syntax can be confusing sometimes. To double-check your cron syntax, you can use an online and free tool called crontab guru. Besides this, the page contains many examples that can help understand the syntax better.

For example, this workflow is configured to run every Tuesday at 6 AM UTC (8 AM CEST) time:

```
on:
schedule:
    - cron: "0 6 * * 2"
```

On GitHub Actions, the time is based on UTC, so you may want to consider this during the setup.

NOTE: On GitHub Actions, the shortest interval you can run scheduled workflows is once every 5 minutes.
If you want to run your job in every pull request or have a different use case, you can check out all the available triggers to run your workflow on the GitHub documentation.

3) Install dependencies and run the script

A GitHub workflow is composed of one or more jobs. A job is a set of steps executed in a certain environment defined on the runs-on. In the example below, the CloudList job runs on the latest ubuntu image. After defining the environment, you need to checkout from a branch, for example the main branch, and decide on the Python version. Finally, you need to add a step to install the dependencies and run your script.

```
jobs:
CloudList:
    runs-on: ubuntu-latest
    steps:
    - name: Checkout the repo
        uses: actions/checkout@v2
    - name: Set up Python 3.8
        uses: actions/setup-python@v2
        with:
        python-version: "3.8"
    - name: Install dependencies
        run: |
        python -m pip install --upgrade pip
        pip install -r requirements.txt
    - name: Update available cloud list
        run: make cloud-list
```

With this, the job already runs periodically and generates the file. However, the final goal is to open a pull request so maintainers can check the changes and merge the code. So let's see how you can add the pull request step to your workflow.

4) Create Pull request

You can create a pull request by using the Create Pull request action in our workflow. A cool feature here is that you can add some customization related to your pull request that can help in the pull request review process, including labels and a defined branch name, for example:

```
- name: Create Pull Request
uses: peter-evans/create-pull-request@v4
with:
    commit-message: Cloud - Update available list
    title: Cloud - Update available list
    body: Cloud - Update available list
    base: main
    labels: automated-pr, Clouds & Regions
    branch: cloud-update-advanced-params
    delete-branch: true
```

The workflow runs the specified jobs according to the scheduled time even if the action is not merged.

Besides periodically running a workflow, you can combine it with the option to manually trigger your workflow, which can help you to test it. You can check out this GitHub Actions manual for that.

When the action takes place, this is what the shiny new pull request looks like ✨:

So every week, the workflow runs and keeps our Aiven cloud list always updated for our beloved users. 🧡

Conclusion

As devportal maintainers, we are happy to use GitHub Action in our CI/CD process to keep our Aiven cloud list always up-to-date.

In this article, we reviewed the workflow code one segment at a time. The complete script looks like this:

File: .github/workflows/cloud-list.yaml

name: Cloud - Create PR to update available list
on:
  schedule:
    - cron: "0 6 * * 2"
jobs:
  CloudList:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout the repo
        uses: actions/checkout@v2
      - name: Set up Python 3.8
        uses: actions/setup-python@v2
        with:
          python-version: "3.8"
      - name: Install dependencies
        run: |
          python -m pip install --upgrade pip
          pip install -r requirements.txt
      - name: Update available cloud list
        run: make cloud-list
      - name: Create Pull Request
        uses: peter-evans/create-pull-request@v4
        with:
          commit-message: Cloud - Update available list
          title: Cloud - Update available list
          body: Cloud - Update available list
          base: main
          labels: automated-pr, Clouds & Regions
          branch: cloud-update-advanced-params
          delete-branch: true

Besides periodically running a workflow, you can also add the option to trigger your workflow manually. You can check out this GitHub Actions manual with all the information.

If what you have in mind for your pull request looks different, you can check out more examples on how to customize the create-pull-request actions to fit your use case.

Here you can find more examples to learn how to use GitHub Actions to open pull requests for you. Have fun automating your pull requests ⚙️!

I'm a Cloud Engineer at Nordcloud.

Visiting Google’s Office

Laysa Uchoa — Fri, 19 Aug 2022 07:34:25 +0000

The Google’s Munich Office

Google is the house of innovation and a dream company of many folks, especially in TECH. I heard a lot about Google's office and how nice they are. In this post, I will share with you my impressions of Google's Munich office.

The visit

To visit Google's office, you need to be invited by someone who works at the company or attend an event happening in the company. You will receive an identification ID to walk into the building. I was invited by a software engineer from Google who was responsible for me and accompanied me during the whole visit. This was great because I would get lost in this huge building without guidance.

Stats

I heard that this office has approximately 500-700 employees. Also, 80% of employees work in software development. Those are the following areas which those teams are working on:

Chrome tool
Security and Privacy
Engineering and productivity tools
Internal tools

The building

The building is H U G E. The engineer who was accompanying me told me that took him three months to stop getting lost in the building. I was lucky to have someone with me to show me the way.

If you are wondering where is the office, Google's Munich office is located at Erika-Mann-Straße 33. If you are driving by public transportation, the nearest metro station is the München Donnersbergerbrücke. It's a very central location! Now, let's talk about the office's delicious food...

The food

There are two restaurants in the building. The food is of high quality. In the menu, we had fried chicken, fish, soya balls, and salads. Also, you could get your own wok plate. Everything was very fresh and prepared in front of us.

And because it is summer, we even had the chance to have ice cream 🍦

Besides the restaurants, Google's Munich office has its own rooftop Cafe. Not only the coffee was delicious, but we also had Käseschmarrn and some Tiramisu there. Also, you have access to nice drinks like tea, juices, beer, and a variety of snacks too.

You know what they say

Out of sight, out of mind

So imagine you can only see the healthy options, would it make it easy for you to make better decisions?! Google's office design takes this to another level. The refrigerators are designed to show healthy choices and hide the unhealthy ones.

Rooms and Office Decoration

The building has many rooms for all tastes. Some rooms are inspired by the local metro stations like the Candidplatz room, for example, inspired in the U-Bahnhof Candidplatz.

In the Munich office, you can also find Obatzda room named after a Bavarian delicacy, Marienplatz named after the central square of Munich and they even have their own Google Straße which means Google Street in German.

What do you think? I love the idea!!! ❤️

Also, the building has room for almost everything one can wish for:

Sleep and nap 😴
Massage and relax 🧘🏻‍♀️
Breastfeed 🤱
Biophilia room with lots of plants 🌱
Video games and boarding games 🕹
Library room 📚
Cooking room 🥨
Music room to learn piano, guitar, etc 🎸
Fitness studio 🏃🏻‍♀️

It REALLY feels like you are in some sort of paradise! 🤩

Sustainability

The building recycles 2.409 million litres of rainwater per year. Also, each open office and pretty much everywhere has trash cans with a separation system.

Seems simple but I've never been in an office where also the open office had such a thing. 🤔

One interesting thing is that if you need anything hardware, you can buy it directly from a machine located in Google's office, so no need to ask for someone to buy it for you and wait many days until it arrives.

There is also a place with free things that the employees leave there for other employees to take. Who does not love free stuff?! I even got a nice keyboard. I find very sustainable and maybe an idea that other offices can use ;)

Conclusion

Visiting such an office was a joy! I hope you enjoyed the text and if you have any questions about Google's office, feel free to drop here!

Hey, let's connect! ♡
You can follow me on Twitter or add me on LinkedIn.

Laysa UchoaFollow

grateful for helpful compiler messages. Working as Cloud Engineer at Nordcloud.

Write search queries with Python and OpenSearch®

Laysa Uchoa — Mon, 01 Aug 2022 10:44:27 +0000

A Python-perfect dinner party with OpenSearch®

Introduction

When I plan a dinner party, I want my guests to have a great experience, and definitely, I do not want anyone hungry. I need to check the ingredients, my guests' diet restrictions, and preferences. If you also feel that planning that special dinner can be challenging, you are in for a treat.

In this blog, I'll show how to find delicious recipes in the pythonic way. We just need data, Python and the powers of OpenSearch® to plan a perfect dinner party!

Our support material in this learning journey will be a useful CLI application that lets you explore common types of OpenSearch query, and even run them yourself.

Getting started

Some may agree that our search results can be only as good as our dataset. But do not worry, we will be using a high-quality dataset from Epicurious which contains over 20.000 full recipes, ratings, and nutrition information for us.

I'll be using Aiven's fully managed OpenSearch service to get our cluster up and running. I've prepared a demo that contains all the code to connect, send data and perform the search queries.

Everything is explained in the project README.rst, so you can just focus on understanding the queries.

Ingest data to the OpenSearch cluster

The first step is to load the data into our OpenSearch cluster, so we can start to query. Check out how to load the data to your cluster using the Python client.

Now, we can start to play with data!

Finding recipes

Range query

Rumors are that my grandfather is coming to the dinner, so I know I should have at least one recipe that is low in sodium. Low sodium meals are recommended to reduce blood pressure, but in general, this is a good healthy option for everyone.

We can use the range function to help us to find documents where the field value (in this case, sodium) is within a certain range.

Recipes under 140 ms of sodium per serving are considered low sodium meals. So let’s look for recipes around 100 - 140 mg, and build this query as:

{
   "query":{
      "range":{
         "sodium":{
            "gte":100,
            "lte":140
         }
      }
   }
}

We can use the demo program to see the range query in action by running:

python search.py "sodium" 100 140

I'm curious to see what kind of recipes, we get under this condition, and here we go:

['Salsa Verde ',
 'Green Bean and Red Onion Salad with Warm Cider Vinaigrette ',
 'Toasted-Pecan Pie ',
 'Provençal Chicken and Tomato Roast ',
 'Sauteed Cod Provençale ',
 'Roasted Potatoes and Asparagus with Parmesan ',
 'Sweet-and-Sour Baby Carrots ',
 'Ricotta Puddings with Glazed Rhubarb ',
 'Butternut Squash and White Bean Soup ',
 'Turkish Zucchini Pancakes ']

'Turkish Zucchini Pancakes' seems like a delicious recipe, so this would be my choice.

Match-phrase query

Also, I need to find a delicious salad recipe for the occasion 🥗. It's radish season and this vegetable goes really well in summer salads, so let's use match_phrase to look for a "title" containing "Salad with Radish".

{
   "query":{
      "match_phrase":{
         "title":{
            "query":"Salad with radish"
         }
      }
   }
}

We can run this query using the demo program:

python search.py match-phrase "title" "Salad with Radish"

and here is our result:

['Green Bean and Red Onion Salad with Radish Dressing ']

We only got one match and it seems like radish is only part of the dressing. The reason is that the order of words is important when you use match_phrase. In this case, the phrase Salad with Radish only appeared once, hence our single result.

We can fix that by adding some flexibility to our search. There is a powerful feature on match_phrase that allows us to define the distance that the search words can be from each other. This parameter is called slop (default=0). So let's try again with the slop parameter set to 3.

{
   "query":{
      "match_phrase":{
         "title":{
            "query":"Salad with radish",
            "slop":3
         }
      }
   }
}

We can run this query using the demo program:

python search.py match-phrase "title" "Salad with Radish" --slop 3

Not surprisingly, we got more results this time:

['Green Bean and Red Onion Salad with Radish Dressing ',
 'Winter Salad with Black Radish, Apple, and Escarole ',
 'Avocado Radish Salad with Lime Dressing ',
 'Chickpea Salad Sandwich With Creamy Carrot-Radish Slaw ',
 'Mâche, Frisée, and Radish Salad with Mustard Vinaigrette ',
 'Frisée and Radish Salad with Goat Cheese Croutons ',
 'Endive, Mâche, and Radish Salad with Champagne Vinaigrette ',
 'Butter Lettuce and Radish Salad with Fresh Spring Herbs ',
 'Butter Lettuce and Radish Salad with Lemon-Garlic Vinaigrette ',
 'Shaved Carrot and Radish Salad With Herbs and Pumpkin Seeds ']

Now, our results match with "Radish Salad with", "Salad with <something else> Radish" and so on.
We can pick one and move forward to find a desert.

Match query

Let's explore how the match function works, building a query to find "Chocolate Carrot Cake" in our "title".

{
   "query":{
      "match":{
         "title":{
            "query":"Chocolate Carrot Cake",
            "operator": "and"
         }
      }
   }
}

The match parameter will report results in a sorted order of how closely they relate to "Chocolate Carrot Cake" 🥕. By default match uses the "OR" operator, giving results for "Chocolate" or "Carrot" or "Cake". However, I want to have all these terms included in the "title" when we search. We can use the "AND" operator for that:

python search.py match "title" "Chocolate Carrot Cake" --operator "and"

Here are our cake results.

['Chocolate-Orange Carrot Cake ',
 'Milk Chocolate Semifreddo with Star Anise Carrot Cake ']

Everything seems delicious and we are ready for the party 🥳!

And what's for your dinner? You can play around writing your own search queries and find your own perfect dinner.

I'll be happy to hear from you or answer any questions you may have at laysa.uchoa@gmail.com.

Happy meal, everyone!

Examples and other resources

PyCamp 2022 - Going camping without bugs

Laysa Uchoa — Thu, 05 May 2022 07:02:20 +0000

What it is
Where it is
PyCamp activities
Who can join
Final thoughts

What is the PyCamp

🌱 Nature + 🤗 OSS + 🐍 Python friends = ⛺ PyCamp

After many years of PyCamp events in Argentina, the event had its first edition in Europe this Spring in Barcelona, Spain.

Our plan is to not have a fixed plan.

I heard this from the organizer Manuel, @reydelhumo. And you all must be thinking...

Yes! So how a non business event was so organized and successful? The answer is community.

Where to camp

The location chosen was the beautiful region of Girona near Barcelona, Spain. Surrounded by nature, the pythonistas could choose any project or workshop to participate in. What was the view from there? Breath and take a look!

Any similarity with Window desktop image is just pure coincidence...

We did not sleep in tends in the event. We lived in a very well equipped house with shared rooms and bathrooms. Also, with enough sockets to charger our laptops.

PyCamp Activities

The event counted with 25 people spending 4 days together doing what they love the most - collaborating with others through code.

From translating Japanese mangas to building bots and making music with Python, the projects were interesting and fun. Some of themes were: Python packaging, Twitter API, Telegram bots and more.

I worked in one of the projects named as Pylículas. The goal was to make a Telegram bot that could give movie recommendations to the user. Fun!

Also, if you did not find any projects that you like, you could always propose one yourself even if you do not know how to do. People are there are very collaborative and helpful.

Who can join

If you like Python and want to contribute to OSS projects, this is the right event for you. Amazing event, right? But you are not a Pythonista yourself? No problem at all.

In this edition, we had people talking about DevOps tools and JavaScript frameworks, so feel free to bring different topics to the event. Python beginners, we hear you! There are a bunch of mentors to help you during the projects.

Final thoughts

So my final words about this event is that living and collaborating with others makes without saying a true community event. So see you next Spring at PyCamp Spain? ⛺

Copy OpenSearch index data to S3

Laysa Uchoa — Sun, 20 Mar 2022 13:16:00 +0000

It is a good practice to perform backups of your OpenSearch (OS) data to another storage service. This way you can access your data and restore it if something unexpected happens to it.

In this article, you can find out how to dump your OpenSearch data to
an:

AWS S3 bucket

To copy the index data, we will be using elasticsearch-dump
tool. You can read the instructions on GitHub on how to install it. From this library, we will use elasticdump command to copy the input index data to a specific output.

Make sure to have elasticsearch-dump tool installed for the next steps.

Copying data from OpenSearch to AWS S3

Prerequisites

OpenSearch cluster as the input
AWS S3 bucket as the output

Information about your OS cluster and AWS service:

OpenSearch:

SERVICE_URI: OpenSearch service URI.
INPUT_INDEX_NAME: the index that you aim to copy from your input source.

S3 bucket:

AWS credentials (ACCESS_KEY_ID and SECRET_ACCESS_KEY).
S3 file path. for e.g. s3://${BUCKET_NAME}/${FILE_NAME}.json

Find more information about AWS credentials in the AWS docs.

Export OpenSearch index data to S3

Use elasticsearch-dump command to copy the data from your OpenSearch cluster to your AWS S3 bucket. Use your
OpenSearch SERVICE_URI for the input. For the output, choose an AWS S3 file path including the file name that you want for your document.


 {.shell}
elasticdump \
--s3AccessKeyId "${ACCESS_KEY_ID}" \
--s3SecretAccessKey "${SECRET_ACCESS_KEY}" \
--input=SERVICE_URI/INPUT_INDEX_NAME --output "s3://${BUCKET_NAME}/${FILE_NAME}.json"

That is how you can copy your OpenSearch data to a S3 bucket. 🙋🏻‍♀️

print("Hello, devs!")

Laysa Uchoa — Sun, 20 Mar 2022 12:54:57 +0000

Hello, peeps!

My name is Laysa Uchoa and I am brazilian living in Munich, Germany. I am a software developer curious about structured and unstructured data with a mission to share helpful things I find while working in tech. Let's connect! 🐦

I love Python and Python communities. So few years ago, I started organizing Pyladies Munich to offer this space for coding ladies to meet and connect with each other.

I really believe that: If you can dream, you can make it happen in Python 🐍

Besides coding and running tech conferences, you can find me learning new languages. I love human languages (Portuguese 🇧🇷, Spanish 🇪🇸, German 🇩🇪 and English 🇺🇸) besides programming languages.

Your turn: drop "Hi" in your programming or human language, and I will try to guess 🔎

How to migrate your Elasticsearch client to OpenSearch

Laysa Uchoa — Sun, 20 Mar 2022 12:03:04 +0000

In this article, I will show you how to switch from Elasticsearch to an OpenSearch client.

When we think about database migration, we normally think about migrating the data itself. However, you have to consider migrating your client, and especially which version is being used.

In this article, we will briefly explain the database migration process, and how to perform your client migration.

Client migration from Elasticsearch to OpenSearch

Migrating a native Elasticsearch client to OpenSearch requires changes in the client code so that you can continue to interact with your cluster. Are you curious how this can be done? Here we will go over client migration in three languages: Python, Java, and JavaScript (Node.js).

Before diving into the code, there is one thing you should keep in mind. According to OpenSearch docs, Elasticsearch clients for v7.10.2 should work in terms of compatibility with OpenSearch client v1. However, the latest version of Elasticsearch clients may contain checks that can break compatibility. Here are the recommendations regarding the Elasticsearch client version that you should have to migrate to OpenSearch v1.0.0.

Client	Recommended version
Java low-level REST client	7.13.4
Java high-level REST client	7.13.4
Python	7.13.4
NodeJS	7.13.0

Check the full table on the OpenSearch page.

As you can see, it is recommended that first, you upgrade or downgrade your Elasticsearch to match v7.13.4 or v7.13.0, check if is still running correctly, then migrate to the compatible OpenSearch version v1.0.0. Finally, you can upgrade to the latest OpenSearch version that contains additional features and bug fixes. Doing this will help you correct API incompatibilities that may appear during your client migration process.

So let's check how those changes are done code-wise.

Python

For Pythonists, the changes needed in their Python client are regarding the library in use, and how their Python client objects are called. Here we are considering the official Python client libraries for Elasticsearch and OpenSearch.

With a few steps, you can replace your Elasticsearch client with the OpenSearch one.

In the dependencies, change libraries and versions:

- elasticsearch==7.10.2
+ opensearch-py==1.0.0

In the source code, change the imports:

- from elasticsearch import Elasticsearch
+ from opensearchpy import OpenSearch

and the client:

- client_against_opensearch = Elasticsearch(ES_SERVICE_URI, use_ssl=True)
+ client_against_opensearch = OpenSearch(OS_SERVICE_URI, use_ssl=True)

The good news is that you can reuse the same APIs as the Elasticsearch ones in your OpenSearch client. Take a look at the full example on our OpenSearch migration repository.

If you want to know more about the Python OpenSearch client and its compatibility, feel free to explore those resources:

Java

Good news for Java client users, minimal changes are needed when you are migrating to OpenSearch. You only need to install the new dependencies and change your imports. Here you can find the changes related to imports:

- implementation 'org.elasticsearch.client:elasticsearch-rest-client:7.10.2'
- implementation 'org.elasticsearch.client:elasticsearch-rest-high-level-client:7.10.2'
+ implementation 'org.opensearch.client:opensearch-rest-client:1.1.0'
+ implementation 'org.opensearch.client:opensearch-rest-high-level-client:1.1.0'

In your Java client, the changes look like this:

- private static final String CLIENT_LIBRARY = "org.elasticsearch.client:elasticsearch-rest-client:7.10.2";
+ private static final String CLIENT_LIBRARY = "org.opensearch.client:opensearch-rest-client:1.1.0";

By changing the imports, you should be able to use your OpenSearch client with the same APIs as the Elasticsearch ones.

Find out more about OpenSearch Java client, including a full running example of OpenSearch client migration on the links below:

JavaScript (Node.js/NodeJS)

Client migration in JavaScript is pretty straightforward, you only need to install the new dependency and change the require statement.

The dependencies can be installed with npm as follows:

$ npm install --save @opensearch-project/opensearch

You can find in this sample the client change for the NodeJS client:

- const { Client } = require('@elastic/elasticsearch');
+ const { Client } = require('@opensearch-project/opensearch');

You should be able to reuse the APIs once the imports are set correctly.

Check more resources for OpenSearch migration with NodeJS client:

Resources

The examples in this article are part of the OSS project for OpenSearch migration examples, feel free to make a fork and add more examples in your favorite languages. It is a nice way to share your knowledge with the OpenSearch and Elasticsearch communities.

DEV Community: Laysa Uchoa

🔐 Google Cloud Authentication with Workload Identity Federation for GitHub Actions

Introduction

Steps to Set Up Workload Identity Federation

Step 1: Create a Workload Identity Pool

Step 2: Configure the Workload Identity Pool Provider

Step 3: Configure IAM Roles

Step 5: Full Terraform code

Step 6: Setup GitHub Actions Workflow

Step 7: GitHub Action in Action

More Ressources

Get OpenSearch mapping with Python

A Decade of Docker! 🐋

The Birth of Docker

The Birth of "Docker"

Docker's Journey Unfolds

A Developer's Dream

First World Appearance of Docker

Happy learning! 👋

Automating Terraform Deployments with GitHub Actions: A Step-by-Step Guide

Introduction

Scenario

Setup

1) Configure GitHub Actions trigger and permissions

2) Add initial Setup

3) Check the Terraform code

4) Add Terraform steps

5) Create Issue for approval

Conclusion

Automate GitHub Pull Requests with GitHub Actions

Scenario

Configure GitHub Actions workflow

1) Check the script

2) Create a GitHub workflow

3) Choose a trigger

3) Install dependencies and run the script

4) Create Pull request

Conclusion

Read more

Visiting Google’s Office

The Google’s Munich Office

The visit

Stats

The building

The food

Rooms and Office Decoration

Sustainability

Conclusion

Laysa UchoaFollow

Write search queries with Python and OpenSearch®

A Python-perfect dinner party with OpenSearch®

Introduction

Getting started

Ingest data to the OpenSearch cluster

Finding recipes

Range query

Match-phrase query

Match query

Examples and other resources

PyCamp 2022 - Going camping without bugs

Table Of Contents

What is the PyCamp

Where to camp

PyCamp Activities

Who can join

Final thoughts

Copy OpenSearch index data to S3

Copying data from OpenSearch to AWS S3

Prerequisites

Export OpenSearch index data to S3

print("Hello, devs!")

How to migrate your Elasticsearch client to OpenSearch

Client migration from Elasticsearch to OpenSearch

Python

Java

JavaScript (Node.js/NodeJS)

Resources