DEV Community

Hiroshi Toyama
Hiroshi Toyama

Posted on

Trace Sinks Are Gone. Here Is the Terraform That Replaces Them.

If you still export Cloud Trace to BigQuery through a sink, that pipeline is already dead. Per the Trace release notes, trace sinks were deprecated on February 18, 2026, and since August 31, 2026 you can no longer create or edit one. The replacement is Observability Analytics: a linked dataset that exposes the trace storage to BigQuery with no copy of the data.

The same mechanism exists for Cloud Logging, and that is where it gets confusing. Both features are called "linked dataset", both produce a BigQuery dataset holding a view, and both are configured with Terraform. Almost everything else about them differs. I wired up both in one project, and this is what actually had to be true.

Two bucket systems that look like one

The single biggest time sink: _Trace is not a Cloud Logging log bucket.

$ gcloud logging buckets list --project=my-project
LOCATION  BUCKET_ID  RETENTION_DAYS  LIFECYCLE_STATE
global    _Default   30              ACTIVE
global    _Required  400             ACTIVE
Enter fullscreen mode Exit fullscreen mode

No _Trace. It is not hidden, it is not a permissions problem — it is in a different system entirely:

$ gcloud observability buckets list --location=us --project=my-project
name: projects/my-project/locations/us/buckets/_Trace
description: Bucket for storing spans from Cloud Trace.
Enter fullscreen mode Exit fullscreen mode

Note the location. Log buckets default to global; observability buckets follow the project's observability default storage location, which here is us. Any blog post or LLM answer telling you to point google_logging_project_bucket_config at _Trace in global is wrong on both counts.

Cloud Trace Cloud Logging
Storage observability bucket _Trace, dataset Spans log bucket _Default
Location project's observability default (e.g. us) global
CLI gcloud observability buckets gcloud logging buckets
Enable step none — the dataset already exists enable_analytics = true
Terraform google_observability_link google_logging_project_bucket_config + google_logging_linked_dataset
View _AllSpans _AllLogs

While we are listing things that do not exist: there is no google_cloud_trace_sink resource. Ask an LLM for Terraform to export Cloud Trace to BigQuery and there is a good chance it invents one, because the API-level concept existed right up until it was turned off.

Cloud Trace

Terraform support for observability buckets and links landed on September 22, 2026, in provider 8.2.0. The link itself is four fields:

data "google_project" "this" {
  project_id = var.project_id
}

resource "google_project_iam_member" "observability_bigquery" {
  project = var.project_id
  role    = "roles/bigquery.admin"
  member  = "serviceAccount:service-${data.google_project.this.number}@gcp-sa-observability.iam.gserviceaccount.com"
}

# The grant needs time to propagate before the link is created.
resource "time_sleep" "wait_observability_iam" {
  create_duration = "90s"
  depends_on      = [google_project_iam_member.observability_bigquery]
}

resource "google_observability_link" "trace" {
  project  = var.project_id
  location = "us"      # must match the _Trace bucket, NOT global
  bucket   = "_Trace"
  dataset  = "Spans"
  link_id  = "trace_linked_dataset"   # becomes the BigQuery dataset name

  depends_on = [time_sleep.wait_observability_iam]
}
Enter fullscreen mode Exit fullscreen mode

The IAM grant is the part people skip. The dataset on the BigQuery side is created and managed by the Observability service agent, so without roles/bigquery.admin on service-<PROJECT_NUMBER>@gcp-sa-observability.iam.gserviceaccount.com the link creation fails. The service agent already exists if trace data has ever landed in the project; confirm with:

gcloud observability settings describe --location=global --project=my-project
# serviceAccountId: service-123456789012@gcp-sa-observability.iam.gserviceaccount.com
Enter fullscreen mode Exit fullscreen mode

link_id only accepts letters, digits and underscores — no hyphens, which is not the usual GCP naming convention and will bite you if you mirror your other dataset names.

Cloud Logging

Log buckets need Log Analytics switched on first, and _Default already exists, so this resource acquires it rather than creating it:

resource "google_logging_project_bucket_config" "default" {
  project        = var.project_id
  location       = "global"
  bucket_id      = "_Default"
  retention_days = 30          # match the current value or you will see a diff

  enable_analytics = true
}

resource "google_logging_linked_dataset" "default" {
  parent      = "projects/${var.project_id}"   # not `project`
  location    = "global"
  bucket      = google_logging_project_bucket_config.default.bucket_id
  link_id     = "logging_linked_dataset"
}
Enter fullscreen mode Exit fullscreen mode

Three traps packed into ten lines:

  • enable_analytics cannot be disabled once enabled. There is no undo on this one. Decide before you apply.
  • _Default and _Required are auto-created and cannot be deleted. Terraform takes over the existing bucket on create, and terraform destroy only drops it from state — the bucket and its logs stay. Harmless, but it means retention_days is a live setting on a bucket that already has your logs in it, not a greenfield value.
  • google_logging_linked_dataset takes parent, not project. It is the odd one out among Logging resources, and the error is a flat An argument named "project" is not expected here.

The 403 that creates the resource anyway

First apply, with Application Default Credentials from gcloud auth application-default login:

Error: Error waiting to create Link: googleapi: Error 403: Your application is
authenticating by using local Application Default Credentials. The
observability.googleapis.com API requires a quota project, which is not set by
default.
...
"consumer": "projects/32555940559",
"reason": "SERVICE_DISABLED"
Enter fullscreen mode Exit fullscreen mode

That consumer project number is not yours — it is Google's shared gcloud OAuth client. The Observability API refuses to bill the call to it and demands an explicit quota project. The Terraform-native fix is on the provider, not in your shell:

provider "google" {
  project = var.project_id

  billing_project       = var.project_id
  user_project_override = true
}
Enter fullscreen mode Exit fullscreen mode

Here is the nasty part. The POST succeeded; only the long-running-operation poll got the 403. So the link exists in GCP while Terraform recorded a failure and wrote nothing to state:

$ gcloud observability buckets datasets links list \
    --bucket=_Trace --dataset=Spans --location=us --project=my-project
name: projects/my-project/locations/us/buckets/_Trace/datasets/Spans/links/trace_linked_dataset
Enter fullscreen mode Exit fullscreen mode

Re-running apply would just fail on "already exists". Import it:

terraform import google_observability_link.trace \
  projects/my-project/locations/us/buckets/_Trace/datasets/Spans/links/trace_linked_dataset
Enter fullscreen mode Exit fullscreen mode

Then apply converges to zero changes. Whenever an operation-based GCP resource fails while waiting, check the API before assuming nothing happened.

Querying: --location=US, even for global

Both links produce an ordinary three-part BigQuery path. If you have seen a five-part project.global._Trace.Spans._AllSpans floating around, that is not a thing:

$ bq --project_id=my-project ls
        datasetId
 -----------------------
  logging_linked_dataset
  trace_linked_dataset
Enter fullscreen mode Exit fullscreen mode

The counterintuitive bit is the location. bq --location is the BigQuery job location and has to match the dataset — it has nothing to do with the bucket the data came from. Both links here land in the US multi-region, including the one attached to a global log bucket:

$ bq show --format=prettyjson trace_linked_dataset   | grep location   # "US"  (bucket: us)
$ bq show --format=prettyjson logging_linked_dataset | grep location   # "US"  (bucket: global)
Enter fullscreen mode Exit fullscreen mode

bq resolves the location on its own if you omit the flag, so this is less about remembering to pass US and more about not passing the bucket's location out of habit:

$ bq query --location=global ...
BigQuery error in query operation: Location global does not support this operation.

$ bq query --location=asia-northeast1 ...
Not found: Dataset my-project:trace_linked_dataset was not found in location asia-northeast1
Enter fullscreen mode Exit fullscreen mode

global is not a BigQuery location at all. Read the dataset with bq show instead of assuming the bucket location carries over.

Trace spans, note the OTel-native column names:

SELECT trace_id, span_id, name, start_time,
       duration_nano / 1e6 AS duration_ms,
       status.code AS status_code
FROM `my-project.trace_linked_dataset._AllSpans`
WHERE end_time > TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 1 HOUR)
ORDER BY start_time DESC
LIMIT 100
Enter fullscreen mode Exit fullscreen mode

It is name, not display_name; attributes is a JSON column, not a key-value RECORD; duration is duration_nano. The Cloud Trace v2 API schema that most documentation describes is not what you get in BigQuery — spans arrive in the OpenTelemetry-shaped schema, and a query written against the API docs returns Unrecognized name: display_name.

Logs:

SELECT timestamp, severity, log_name, resource.type AS resource_type,
       COALESCE(text_payload, TO_JSON_STRING(json_payload)) AS payload
FROM `my-project.logging_linked_dataset._AllLogs`
WHERE timestamp > TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 1 HOUR)
ORDER BY timestamp DESC
LIMIT 100
Enter fullscreen mode Exit fullscreen mode

Joining traces to logs

This is the payoff, and the two views disagree on how to spell a trace ID. _AllSpans.trace_id is the bare 32-hex string; _AllLogs.trace is the full resource name projects/<id>/traces/<trace_id>:

SELECT
  s.trace_id,
  s.name AS span_name,
  s.duration_nano / 1e6 AS duration_ms,
  COUNT(l.insert_id) AS log_count
FROM `my-project.trace_linked_dataset._AllSpans` AS s
LEFT JOIN `my-project.logging_linked_dataset._AllLogs` AS l
  ON REGEXP_EXTRACT(l.trace, r"/traces/(.+)$") = s.trace_id
WHERE s.end_time > TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 30 DAY)
GROUP BY 1, 2, 3
ORDER BY duration_ms DESC
Enter fullscreen mode Exit fullscreen mode

Two separate linked datasets, one query, no ETL between them.

What you are actually buying

The cost argument against the old Logging sink is simple: a sink writes a second physical copy into BigQuery, so you pay log bucket storage and BigQuery storage forever. A linked dataset is zero-copy — _AllSpans and _AllLogs are views over the existing bucket, so BigQuery storage cost is zero and you only pay for bytes scanned.

What it does not buy you is retention. Spans in _Trace are kept 30 days, _Default logs 30 days, and the link does not extend either by a single hour. If you need year-over-year analysis, you still need a scheduled query copying into a real BigQuery table — the linked dataset is just a much better source for it than a sink was.


Verified on hashicorp/google 8.11.0, Terraform 1.9.5, gcloud 569.0.0 (gcloud beta observability on that version; it went GA on September 23, 2026).

Top comments (0)