DEV Community

Cover image for GCP Undocumented: Fix Error 403: Permission to delete log notification rules denied
Łukasz Pracki
Łukasz Pracki

Posted on

GCP Undocumented: Fix Error 403: Permission to delete log notification rules denied

#googlecloud #gcp #cloudcomputing #devops

This is a part of a series where I post solutions to problems I solved when working with Google Cloud as a Cloud Engineer. In many cases, I couldn't find those fixes in the Internet myself, and I want to document them to act as a future reference.

I'm currently creating some Cloud Monitoring Alerting Policies using Terraform. The policy we'll discuss today is the one for Service Health.

Google Cloud Service Health

The Service Health page lets you see any active issues with Google Cloud services affecting your workloads.

You can easily create an alerting policy from that page, so you can receive a notification almost instantly, when something happens to Google's infrastructure running your applications.

Here's an example of Terraform code for this alerting policy:

resource "google_monitoring_alert_policy" "service_health_alerts" {
  combiner     = "OR"
  display_name = "Service Health - All incidents, all updates"
  enabled      = true
  notification_channels = [
    google_monitoring_notification_channel.notification-channel-email.name,
  ]
  project     = var.scoping_project
  user_labels = {}

  alert_strategy {
    auto_close           = "1800s"
    notification_prompts = []

    notification_rate_limit {
      period = "300s"
    }
  }

  conditions { [redacted] }
  documentation { [redacted] }
}
Enter fullscreen mode Exit fullscreen mode

The Problem

Somehow all went well when I created the policy with code and applied it. The problem started when I tried to apply any changes to the Terraform resource - changes as simple as switching the policy from enabled = true to enabled = false.

All I was getting with the terraform apply command was the following error:

googleapi: Error 403: Permission to delete log notification rules denied (or the resource may not exist).
Enter fullscreen mode Exit fullscreen mode

This exact error was not documented anywhere.

The Solution

It took me hours to find out but the solution is quite simple.

To create the policy from code, I was using the account that had the Owner role on the monitoring project.

The error came up when the terraform apply command was issued with a GitHub Actions Service Account, which didn't have an Owner role.

The Service Account was a Monitoring AlertPolicy Editor on this project, but that was not enough. To work with a Service Health alert policy, it also needed to be a Logs Configuration Writer on the monitoring project.

Lessons Learned

Here's what I'll take out from this case:

  1. Always verify which account you use to implement changes from the code.
  2. To create and change the alerting policy for Google Cloud Service Health monitoring, your principal needs two roles: Monitoring AlertPolicy Editor and Logs Configuration Writer.

Top comments (0)