DEV Community

AWS Community Builders profile image Aki
Aki for AWS Community Builders

Posted on

Hitting the Iceberg REST Catalog Directly: Understanding the Differences Between Glue Data Catalog and S3 Tables

#aws #iceberg #dataengineering

Original Japanese article: Iceberg REST Catalogを直接叩いて、Glue Data CatalogとS3 Tablesの違いを理解する

Introduction

I'm Aki, an AWS Community Builder (@jitepengin).

Most of the time, when working with Iceberg tables, we reach for PyIceberg or Spark. I'm no exception, and honestly there were parts of the PyIceberg configuration — rest.sigv4-enabled, rest.signing-name, warehouse — that I understood only vaguely.

Iceberg defines a standard called the Iceberg REST Catalog Open API specification, and AWS implements it through two separate endpoints:

  • The AWS Glue Iceberg REST endpoint (https://glue.<region>.amazonaws.com/iceberg)
  • The Amazon S3 Tables Iceberg REST endpoint (https://s3tables.<region>.amazonaws.com/iceberg)

If two implementations follow the same spec, sending the same requests to both and comparing the results should reveal what's actually different between them.

In this article, I'll bypass clients like PyIceberg entirely and hit the REST API directly to explore the differences between the two endpoints.

To state the conclusion up front:

Even though both implement the same Iceberg REST Catalog specification, Glue is designed as an "entry point to multiple catalogs," while S3 Tables is designed as an "entry point to a single table bucket." That difference is visible just by looking at the URL paths.

I previously wrote about the relationship between S3 Tables and Glue Data Catalog in another article — worth a read alongside this one:
Does Amazon S3 Tables Replace AWS Glue Data Catalog? Understanding Their Relationship

What Is the Iceberg REST Catalog?

The Iceberg REST Catalog is a specification that standardizes Iceberg catalog operations as an HTTP API. It's published as an OpenAPI definition (YAML), and any catalog that conforms to it can be accessed the same way from clients such as PyIceberg, Spark, and Trino.

The key points of the spec are:

  • URL paths follow a pattern like GET /v1/{prefix}/namespaces, where {prefix} is a free-form segment
  • Clients first call GET /v1/config to retrieve endpoint configuration (the default prefix and other settings)
  • The table metadata body (schema, snapshots, etc.) is returned as JSON in the LoadTable response

In other words, when you write catalog.load_table("ns.table") in PyIceberg, what's actually happening under the hood is an HTTP request to GET /v1/{prefix}/namespaces/ns/tables/table.

Here's a summary of the two AWS implementations before we dive in:

Item Glue Iceberg REST endpoint S3 Tables Iceberg REST endpoint
Endpoint https://glue.<region>.amazonaws.com/iceberg https://s3tables.<region>.amazonaws.com/iceberg
Contents of {prefix} /catalogs/{catalog} (catalog hierarchy) URL-encoded table bucket ARN
Value passed as warehouse Glue catalog ID Table bucket ARN
SigV4 signing service name glue s3tables
Access control IAM + Lake Formation s3tables IAM actions only

Here's the overall picture as a diagram:

              Iceberg REST Catalog spec
           GET /v1/{prefix}/namespaces/...
                        │
        ┌───────────────┴───────────────┐
        │                               │
  Glue endpoint                  S3 Tables endpoint
        │                               │
 prefix = /catalogs/{catalog}    prefix = table bucket ARN
 (multi-catalog hierarchy)       (one bucket = one catalog)
        │                               │
  IAM + Lake Formation           s3tables IAM actions
        │                               │
        └───────────────┬───────────────┘
                        │
              Same Iceberg table
          (points to the same metadata.json)
Enter fullscreen mode Exit fullscreen mode

Note: a catalog doesn't manage the metadata.json file itself — it provides a reference to where the latest metadata-location is. The catalog's essential job is knowing where the current metadata.json lives.

Setting Up the Test Environment

Let's create a test table bucket, namespace, and table via the CLI. Read the region as Tokyo (ap-northeast-1) and the account ID as 123456789012.

# Create a table bucket
aws s3tables create-table-bucket \
  --name penguin-rest-test \
  --region ap-northeast-1

# Create a namespace
aws s3tables create-namespace \
  --table-bucket-arn arn:aws:s3tables:ap-northeast-1:123456789012:bucket/penguin-rest-test \
  --namespace analytics

# Create a table (with schema)
aws s3tables create-table \
  --table-bucket-arn arn:aws:s3tables:ap-northeast-1:123456789012:bucket/penguin-rest-test \
  --namespace analytics \
  --name daily_sales \
  --format ICEBERG \
  --metadata '{
    "iceberg": {
      "schema": {
        "fields": [
          {"name": "sales_date", "type": "date", "required": false},
          {"name": "amount", "type": "long", "required": false}
        ]
      }
    }
  }'
Enter fullscreen mode Exit fullscreen mode

SigV4 Signing

You can't hit these endpoints with plain curl.

The Iceberg REST Catalog spec defines an OAuth2-based authentication flow, but AWS's implementation uses IAM SigV4 signing instead — a standard-spec API with AWS-flavored authentication.

That rest.sigv4-enabled: true setting in PyIceberg is exactly what enables this signing.

Computing SigV4 signatures by hand is painful, so I used awscurl for this exercise.

pip install awscurl
Enter fullscreen mode Exit fullscreen mode

awscurl picks up credentials from environment variables or a profile and sends SigV4-signed requests for you.

The important part is the --service option, which must specify the correct service name for signing: glue for the Glue endpoint, s3tables for the S3 Tables endpoint.

Hitting the S3 Tables Iceberg REST Endpoint

Let's start with the simpler of the two — the S3 Tables endpoint.

GET /v1/config

getConfig is the first API a REST Catalog client calls. The warehouse query parameter takes the table bucket ARN.

Note

When using awscurl, pass the raw, unencoded ARN as the query parameter value. awscurl automatically URL-encodes query parameters before computing the signature, so if you pre-encode the value yourself, it gets double-encoded and you'll get a SignatureDoesNotMatch error.

BUCKET_ARN="arn:aws:s3tables:ap-northeast-1:123456789012:bucket/penguin-rest-test"

awscurl --service s3tables --region ap-northeast-1 \
  "https://s3tables.ap-northeast-1.amazonaws.com/iceberg/v1/config?warehouse=${BUCKET_ARN}"
Enter fullscreen mode Exit fullscreen mode

Result (excerpt):

{
  "defaults": {
    "prefix": "arn%3Aaws%3As3tables%3Aap-northeast-1%3A123456789012%3Abucket%2Fpenguin-rest-test",
    "io-impl": "org.apache.iceberg.aws.s3.S3FileIO",
    "write.object-storage.enabled": "true",
    "write.object-storage.partitioned-paths": "false",
    "s3.delete-enabled": "false",
    "rest-metrics-reporting-enabled": "false"
  },
  "overrides": {}
}
Enter fullscreen mode Exit fullscreen mode

The interesting part is the prefix in the response. For the S3 Tables endpoint, prefix is exactly the URL-encoded table bucket ARN, and it lives under defaults. In other words, this endpoint is designed around a "one table bucket = one catalog" model — if you want to work with multiple table buckets, you register multiple catalogs on the client side.

defaults also included the write-side FileIO implementation (io-impl), object storage layout settings (write.object-storage.*), and a setting that disables S3 delete operations (s3.delete-enabled). It's interesting to see, from actual output, that some of the parameters we're used to configuring individually on the client (PyIceberg) side are actually being pushed down as server-side defaults through this config response.

Incidentally, this getConfig call is authorized under the s3tables:GetTableBucket IAM action. The official documentation includes a mapping table showing which s3tables IAM action corresponds to each REST operation.

Note

The spec's CatalogConfig also defines an endpoints field, which lets the server return a list of supported endpoints (in a format like "GET /v1/{prefix}/namespaces"). When I checked, the S3 Tables endpoint's response did not include an endpoints field.

Listing Namespaces and Tables

Now that we know the prefix, let's list namespaces and tables.

Note

There's a subtlety worth calling out here.

The warehouse query parameter for GET /v1/config needs the raw ARN, since awscurl automatically URL-encodes query parameters before signing — pre-encoding it yourself causes double-encoding and a SignatureDoesNotMatch error.

The {prefix} segment in the path, however, behaves differently. If you put the ARN into the path completely unencoded (colons and slashes as-is), the / characters inside the ARN get interpreted as path separators, and the request no longer matches the modeled URL pattern /v1/{prefix}/namespaces — you get an UnknownOperationException.

The correct approach is to leave the colons raw but percent-encode only the single / inside bucket/<table-bucket-name> as %2F. That makes the request match the correct route (and once it reaches the authorization layer, you'll get a 403 if you lack permissions, rather than a routing error).

Even though it's the same operation — putting an ARN into a URL — the encoding rules differ between query parameters and path segments. That's something you only really notice by actually hitting the endpoint.

BUCKET_ARN_PATH="arn:aws:s3tables:ap-northeast-1:123456789012:bucket%2Fpenguin-rest-test"

# List namespaces
awscurl --service s3tables --region ap-northeast-1 \
  "https://s3tables.ap-northeast-1.amazonaws.com/iceberg/v1/${BUCKET_ARN_PATH}/namespaces"

# List tables
awscurl --service s3tables --region ap-northeast-1 \
  "https://s3tables.ap-northeast-1.amazonaws.com/iceberg/v1/${BUCKET_ARN_PATH}/namespaces/analytics/tables"
Enter fullscreen mode Exit fullscreen mode

Result (excerpt):

{
  "namespaces": [
    ["analytics"]
  ]
}
Enter fullscreen mode Exit fullscreen mode 
{
  "identifiers": [
    {
      "name": "daily_sales",
      "namespace": ["analytics"]
    }
  ]
}
Enter fullscreen mode Exit fullscreen mode

There's one difference from the spec worth noting here: the Iceberg REST spec allows multi-level namespaces (like a.b.c), but S3 Tables only supports a single level.

Let's confirm this by trying to create a multi-level namespace:

awscurl --service s3tables --region ap-northeast-1 \
  -X POST \
  -H "Content-Type: application/json" \
  -d '{"namespace": ["level1", "level2"]}' \
  "https://s3tables.ap-northeast-1.amazonaws.com/iceberg/v1/${BUCKET_ARN_PATH}/namespaces"
Enter fullscreen mode Exit fullscreen mode

Result (excerpt):

{
  "error": {
    "code": 400,
    "message": "Multipart namespaces are not supported.",
    "type": "bad_request"
  }
}
Enter fullscreen mode Exit fullscreen mode

The error message spells it out directly — "multipart namespaces are not supported" — confirming the single-level restriction empirically.

Retrieving Metadata with LoadTable

This is the API I most wanted to check today.

awscurl --service s3tables --region ap-northeast-1 \
  "https://s3tables.ap-northeast-1.amazonaws.com/iceberg/v1/${BUCKET_ARN_PATH}/namespaces/analytics/tables/daily_sales"
Enter fullscreen mode Exit fullscreen mode

Result (excerpt):

{
  "metadata-location": "s3://34c72c19-610d-4e5c-d8d1qwx3db1p3tbr8g9jch9n6e14sapn1b--table-s3/metadata/00000-163a447a-c64d-44f9-8619-5db9561e549b.metadata.json",
  "metadata": {
    "format-version": 2,
    "table-uuid": "9507ea2b-7a71-40a7-b6a9-1d6e633955e1",
    "location": "s3://34c72c19-610d-4e5c-d8d1qwx3db1p3tbr8g9jch9n6e14sapn1b--table-s3",
    "current-schema-id": 0,
    "schemas": [
      {
        "type": "struct",
        "schema-id": 0,
        "fields": [
          { "id": 1, "name": "sales_date", "required": false, "type": "date" },
          { "id": 2, "name": "amount", "required": false, "type": "long" }
        ]
      }
    ],
    "default-spec-id": 0,
    "partition-specs": [{ "spec-id": 0, "fields": [] }],
    "properties": {
      "write.parquet.compression-codec": "zstd"
    },
    "current-snapshot-id": -1,
    "snapshots": []
  },
  "config": {
    "tableBucketId": "3b2a6702-dd99-44f2-bc03-2f9f6b273104",
    "namespaceId": "20576faa-861e-499f-a30f-4d6c7550dd55",
    "tableId": "34c72c19-610d-4e5c-bee4-6ae9f093c583"
  }
}
Enter fullscreen mode Exit fullscreen mode

This JSON is exactly what's behind the information we normally see through table.schema() or table.snapshots() in PyIceberg. The response directly confirms that a catalog's essential job is "managing and returning the S3 path of the latest metadata.json" (metadata-location).

Incidentally, the config field in the response contained S3 Tables-internal identifiers: tableBucketId, namespaceId, and tableId. Per spec, the Glue endpoint's config is supposed to include temporary credentials (vended credentials), but here on the S3 Tables endpoint there's no credential delegation at all — it just returns internal management IDs.

Note

The S3 Tables endpoint also documents a limit: operations against a table whose metadata.json exceeds 50MB return a 400 error. Worth keeping in mind — if metadata bloats from accumulated snapshots and similar growth, you could hit this API-level ceiling.

Checking What's Happening Under the Hood via CloudTrail

An interesting characteristic of the S3 Tables endpoint is that REST API calls get logged in CloudTrail as their corresponding native S3 Tables actions. Per the official documentation, a single LoadTable call logs both of the following:

  • GetTableMetadataLocation (a management event)
  • A data event corresponding to GetTableData

In other words, the audit log itself reveals that this endpoint is implemented as a proxy that translates the Iceberg REST API into native S3 Tables API calls.

Let's check CloudTrail directly:

aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=EventName,AttributeValue=GetTableMetadataLocation \
  --region ap-northeast-1 \
| jq '.Events[0] | {time: .EventTime, user: .Username, event: .EventName}'
Enter fullscreen mode Exit fullscreen mode

Result (excerpt):

{
  "time": "2026-07-09T13:46:44+00:00",
  "user": "penguin-test-user",
  "event": "GetTableMetadataLocation"
}
Enter fullscreen mode Exit fullscreen mode

Even though we only called LoadTable over REST, CloudTrail logs it under the native S3 Tables event name s3tables:GetTableMetadataLocation, complete with the exact caller (IAM username).

Hitting the Glue Iceberg REST Endpoint

Now let's look at the Glue endpoint, which behaves quite differently.

GET /v1/config 

awscurl --service glue --region ap-northeast-1 \
  "https://glue.ap-northeast-1.amazonaws.com/iceberg/v1/config?warehouse=123456789012"
Enter fullscreen mode Exit fullscreen mode

Result (excerpt):

{
  "defaults": {
    "prefix": "123456789012",
    "header.Content-Type": "application/x-amz-json-1.1",
    "rest.sigv4-enabled": "true",
    "rest.signing-name": "glue",
    "rest.signing-region": "ap-northeast-1",
    "rest-table-scan-enabled": "true",
    "rest-data-commit-enabled": "true",
    "token-refresh-enabled": "false"
  },
  "overrides": {
    "prefix": "catalogs/123456789012"
  }
}
Enter fullscreen mode Exit fullscreen mode

There's a difference from the S3 Tables endpoint worth calling out here: defaults.prefix is "123456789012" (just the account ID), while overrides.prefix is "catalogs/123456789012" (with catalogs/ prepended) — the two values genuinely differ. Per spec, overrides takes precedence over defaults, so the prefix a client should actually use is catalogs/123456789012. It's confirmed, from actual output, that defaults and overrides really can disagree.

The other defaults entries are worth a look too — rest.sigv4-enabled and rest.signing-name map directly onto the PyIceberg configuration items of the same name. rest-table-scan-enabled and rest-data-commit-enabled are presumably feature flags for server-side scan planning and write commits, discussed further below.

Incidentally, this response also didn't include an endpoints field.

What you pass as warehouse for the Glue endpoint is a Glue catalog ID (defaulting to the current account's root catalog if omitted). Where the S3 Tables endpoint's warehouse was "a bucket ARN," here it's "where in the catalog hierarchy to connect."

Prefix Rules: Encoding the Catalog Hierarchy

The Glue endpoint's prefix always takes the form /catalogs/{catalog}. The official documentation defines the following rules for how {catalog} is written:

Target Prefix notation Example REST path
Default catalog of the current account : GET /v1/catalogs/:/namespaces
Default catalog of a specific account Account ID GET /v1/catalogs/123456789012/namespaces
Nested catalog catalog1:catalog2 GET /v1/catalogs/rmscatalog1:db1/namespaces
Nested catalog of a specific account accountId:catalog1:catalog2 GET /v1/catalogs/123456789012:s3tablescatalog:bucket/namespaces

The : notation for the default catalog is confusing at first glance, but it makes more sense once you think of it as "encoding the catalog hierarchy's separator as : instead of the path separator /." It's essentially using the free-form nature of the Iceberg REST spec's prefix to cram Glue's multi-catalog hierarchy into the URL.

Listing Namespaces (Default Catalog) 

awscurl --service glue --region ap-northeast-1 \
  "https://glue.ap-northeast-1.amazonaws.com/iceberg/v1/catalogs/123456789012/namespaces"
Enter fullscreen mode Exit fullscreen mode

Note

The default catalog can be expressed either as a single colon (:) or as the account ID. Here I'm using the catalogs/{accountId} form, which is also what showed up in overrides.prefix in the getConfig response above.

Result (excerpt):

{
  "namespaces": [
    ["hive"],
    ["icebergdb"]
  ]
}
Enter fullscreen mode Exit fullscreen mode

What comes back is the familiar list of Glue databases — the response directly confirms the mapping "Glue database = Iceberg namespace." Notably, S3 Tables-backed databases under s3tablescatalog are not included here. The default catalog only covers Glue databases directly under the account; to see S3 Tables namespaces, you need to explicitly reach them via the nested catalog prefix, described below.

Incidentally, the Glue endpoint has the same single-level namespace restriction as S3 Tables. What looks like a multi-level structure isn't expressed by making namespaces deeper — it's expressed by nesting catalogs. That seems to be a consistent design choice on Glue's part.

Reading an S3 Tables Table Through the Glue Endpoint

This is the heart of today's investigation: reading the same table we read via the S3 Tables endpoint, this time through the Glue endpoint.

S3 Tables with Glue integration enabled is mounted under the s3tablescatalog federated catalog. In PyIceberg, you'd specify warehouse as 123456789012:s3tablescatalog/penguin-rest-test. Applying the earlier prefix-conversion rule, the path becomes:

awscurl --service glue --region ap-northeast-1 \
  "https://glue.ap-northeast-1.amazonaws.com/iceberg/v1/catalogs/123456789012:s3tablescatalog:penguin-rest-test/namespaces/analytics/tables/daily_sales"
Enter fullscreen mode Exit fullscreen mode

Result (excerpt):

{
  "config": {
    "createdBy": "123456789012",
    "s3TableArn": "arn:aws:s3tables:ap-northeast-1:123456789012:bucket/penguin-rest-test/table/34c72c19-610d-4e5c-bee4-6ae9f093c583",
    "ownerAccountId": "123456789012",
    "metadata_location": "s3://34c72c19-610d-4e5c-d8d1qwx3db1p3tbr8g9jch9n6e14sapn1b--table-s3/metadata/00000-163a447a-c64d-44f9-8619-5db9561e549b.metadata.json",
    "format": "ICEBERG",
    "warehouse_location": "s3://34c72c19-610d-4e5c-d8d1qwx3db1p3tbr8g9jch9n6e14sapn1b--table-s3",
    "table_type": "ICEBERG"
  },
  "metadata": {
    "format-version": 2,
    "table-uuid": "9507ea2b-7a71-40a7-b6a9-1d6e633955e1",
    "current-schema-id": 0,
    "schemas": [
      {
        "type": "struct",
        "schema-id": 0,
        "fields": [
          { "id": 1, "name": "sales_date", "required": false, "type": "date" },
          { "id": 2, "name": "amount", "required": false, "type": "long" }
        ]
      }
    ],
    "current-snapshot-id": -1
  },
  "metadata-location": "s3://34c72c19-610d-4e5c-d8d1qwx3db1p3tbr8g9jch9n6e14sapn1b--table-s3/metadata/00000-163a447a-c64d-44f9-8619-5db9561e549b.metadata.json"
}
Enter fullscreen mode Exit fullscreen mode

Note

The nested-catalog prefix notation used above (accountId:s3tablescatalog:bucketName) was derived from the conversion-rule table in the official documentation. Sending an actual request in this format resulted in correct routing, and both authorization and the underlying data checked out.

The metadata-location and table-uuid (9507ea2b-7a71-40a7-b6a9-1d6e633955e1) here match exactly what we retrieved via the S3 Tables endpoint earlier. Different entry point, same underlying entity.

The config contents are worth noting too. In this request — where we didn't set the x-iceberg-access-delegation header — config didn't contain S3 access credentials, only metadata like s3TableArn (the original S3 Tables ARN) and warehouse_location (the actual S3 path). As I'll cover below, one more configuration step was needed before credentials would actually come back.

For the same table: through the S3 Tables endpoint the path is /v1/{bucketArn}/namespaces/analytics/tables/daily_sales, while through the Glue endpoint it's /v1/catalogs/{accountId}:s3tablescatalog:{bucketName}/namespaces/analytics/tables/daily_sales. Physically they point to the same metadata.json, but the path structures are completely different because the two services have entirely different mental models of "what a catalog is." That, I think, is the essential difference between the two endpoints.

Access Control Differences (Lake Formation)

Accessing via the Glue endpoint requires Lake Formation grants in addition to IAM policies (glue:GetCatalog, glue:GetTable, etc.), because S3 Tables tables get registered as Lake Formation resources when Glue integration is enabled.

Further, for an external engine to read the actual data, you need to enable full-table access for external engines in Lake Formation, and allow the IAM role to call lakeformation:GetDataAccess. This mechanism is what issues the temporary credentials known as vended credentials.

In practice, though, this alone wasn't enough. It took explicitly registering the table bucket as a federated Lake Formation resource via aws lakeformation register-resource --with-federation, and setting up a trust policy so the Lake Formation service itself (lakeformation.amazonaws.com) could assume the IAM role used for issuing credentials, before vended credentials actually started being issued. If this registration step is missing, requests with the header described below (even with otherwise-correct IAM policies and Lake Formation grants) just come back with a credential-less config — only s3TableArn and warehouse_location — with no error at all, which makes it an easy thing to miss.

When PyIceberg hits the Glue endpoint, it attaches this header to the request:

x-iceberg-access-delegation: vended-credentials
Enter fullscreen mode Exit fullscreen mode

Per the Iceberg REST spec, this header signals to the server that the client wants credential delegation. However — at least in this environment — as long as the federation registration was complete, calling LoadTable without this header still returned credentials the same way.

In other words, what actually determines whether credentials come back doesn't seem to be the presence of the header, but whether the Lake Formation federation registration is complete. The header is the spec-compliant way to signal intent, but at least in this environment it wasn't the deciding factor in the Glue endpoint's behavior.

Still, to verify things properly per spec, let's call LoadTable with the header attached:

# With the vended-credentials header
awscurl --service glue --region ap-northeast-1 \
  -H "x-iceberg-access-delegation: vended-credentials" \
  "https://glue.ap-northeast-1.amazonaws.com/iceberg/v1/catalogs/123456789012:s3tablescatalog:penguin-rest-test/namespaces/analytics/tables/daily_sales"
Enter fullscreen mode Exit fullscreen mode

Result (excerpt):

{
  "metadata-location": "s3://34c72c19-610d-4e5c-d8d1qwx3db1p3tbr8g9jch9n6e14sapn1b--table-s3/metadata/00000-163a447a-c64d-44f9-8619-5db9561e549b.metadata.json",
  "metadata": { "...": "..." },
  "config": {
    "s3.access-key-id": "(masked)",
    "s3.secret-access-key": "(masked)",
    "s3.session-token": "(masked)",
    "s3.session-token-expires-at-ms": "1783651228000",
    "s3TableArn": "arn:aws:s3tables:ap-northeast-1:123456789012:bucket/penguin-rest-test/table/34c72c19-610d-4e5c-bee4-6ae9f093c583",
    "warehouse_location": "s3://34c72c19-610d-4e5c-d8d1qwx3db1p3tbr8g9jch9n6e14sapn1b--table-s3"
  }
}
Enter fullscreen mode Exit fullscreen mode

s3.access-key-id, s3.secret-access-key, and s3.session-token are actually present here — these are the vended credentials. Along with s3.session-token-expires-at-ms (session expiration in epoch milliseconds), this confirms from the response itself that the catalog is issuing a temporary storage key directly.

The S3 Tables endpoint, meanwhile, has no notion of Lake Formation at all — authorization there is handled entirely via s3tables:* IAM actions, and resource-based policies on the table bucket are also available.

Same underlying table, but the authorization model changes depending on which entry point you go through — something worth keeping in mind during access-design work. Incidentally, exactly how far you can deliberately vary Lake Formation permissions and what happens as a result (does removing a grant actually deny access, etc.) feels like enough material for its own article, so I'll leave that for another time.

Summarizing the Differences Between the Two Endpoints

Here's what actually turned up from hitting both endpoints directly.

Basic Structure

Item Glue endpoint S3 Tables endpoint
Endpoint glue.<region>.amazonaws.com/iceberg s3tables.<region>.amazonaws.com/iceberg
Signing service name glue s3tables
warehouse value Glue catalog ID Table bucket ARN
prefix /catalogs/{catalog} (hierarchy encoding) URL-encoded bucket ARN
Catalog scope Account-wide (multi-catalog hierarchy) Single table bucket
Storage targeted General-purpose S3 Iceberg + S3 Tables S3 Tables only
Access control IAM + Lake Formation (hybrid possible) s3tables IAM actions only
Credential vending Vended credentials (requires register-resource --with-federation) None (uses the caller's own credentials)

Where to Use Which, and Gotchas

The official documentation gives guidance on when to use each endpoint:

  • If you only need basic read/write access to a single table bucket: the S3 Tables endpoint
  • If you need to integrate multiple catalog sources, or need centralized governance and fine-grained access control via Lake Formation: the Glue endpoint

From actually hitting both directly, here are a few things I'd flag:

  • Getting the signing service name wrong (glue / s3tables) results in a 403. This is exactly the kind of mistake a wrong rest.signing-name in your client config produces.
  • Same table, different authorization model depending on which endpoint you go through (whether Lake Formation is involved). Be explicit up front about which path you're connecting through when designing permissions.
  • CTAS isn't supported on either endpoint. You can work around this by splitting it into CREATE TABLE + INSERT INTO.
  • dropTable on the S3 Tables endpoint requires purge=true. Depending on your Spark version, DROP TABLE PURGE can end up sending purge=false anyway — in that case, you'll need to delete via the native DeleteTable API instead.

Looking Ahead (Some Personal Thoughts)

Hitting both endpoints directly left me with the impression that each service's design philosophy shows through in how it uses the "free-form" parts of the Iceberg REST Catalog spec.

The S3 Tables endpoint — with its prefix being the bucket ARN, authorization handled through s3tables IAM actions, and CloudTrail logging it as a native API — looks, from this angle, like a thin translation layer over the storage API.

The Glue endpoint, on the other hand, encodes a catalog hierarchy into the prefix, has Lake Formation stepping into authorization, and (when configured correctly) issues temporary storage keys via vended credentials. Both general-purpose S3-based Iceberg and S3 Tables are visible through the same entry point — which lines up with something I speculated in an earlier article, that Glue Data Catalog may be evolving into a metadata plane for AWS as a whole. That same idea seems to show up directly in how the REST API's paths are designed.

The fact that authentication is SigV4 rather than the OAuth the spec assumes is another data point suggesting that how a vendor uses the "free" parts of a standard spec reveals something about its design thinking — probably not unique to the Iceberg REST Catalog.

One more thing I noticed while reading the spec: the latest version defines server-side scan planning endpoints (planTableScan and friends), where the server does the work of building a scan plan. Glue already has a separate extension endpoint (https://glue.<region>.amazonaws.com/extensions) that independently offers server-side scan planning for Redshift Managed Storage — so it looks like a capability that existed as a proprietary extension is now being absorbed into the standard spec. That's an interesting trend — a proprietary extension leading the way before the standard catches up — and I'd like to compare Glue's extension API against the standard plan-related endpoints in a future article.

Snowflake's Catalog-Linked Database (CATALOG_API_TYPE = AWS_GLUE) connects to exactly this Glue endpoint, so seeing how this API gets called from the Snowflake side is another topic I want to dig into going forward.

Conclusion

In this article, I hit the Iceberg REST Catalog directly to sort out the differences between the Glue Data Catalog and S3 Tables endpoints. It took some effort, but I came away with a clearer understanding and a few new insights.

To summarize:

  • Design philosophy: even though both implement the same Iceberg REST Catalog spec, Glue is an "entry point to multiple catalogs" while S3 Tables is an "entry point to a single table bucket" — a difference visible just from the URL paths.
  • Prefix design: the spec's free-form {prefix} is used by Glue to encode a catalog hierarchy (/catalogs/{catalog}), and by S3 Tables to encode the table bucket ARN.
  • Authentication: both use IAM SigV4. Getting the signing service name wrong (glue / s3tables) results in a 403.
  • Authorization model: for the same table, the Glue endpoint goes through IAM + Lake Formation, while the S3 Tables endpoint uses only s3tables IAM actions — the model changes depending on the path.
  • Vended credentials: temporary credentials aren't issued just by attaching the x-iceberg-access-delegation: vended-credentials header — you also need to complete resource registration via register-resource --with-federation.
  • Debugging tips: inspecting the actual CanonicalRequest and headers with awscurl -v, using CloudTrail's errorCode to triage error types, and pre-checking URL assembly with echo were all useful, unglamorous techniques.

Day to day, relying on PyIceberg's or Spark's abstractions is more than enough. But looking at the raw HTTP requests once gives you a clearer mental map between each client configuration item and "that part of that request," which raises the resolution you get when triaging connection errors.

I hope this article is useful to anyone trying to understand how the Iceberg REST Catalog actually works.

Top comments (0)