DEV Community

AWS Community Builders profile image Aki
Aki for AWS Community Builders

Posted on

Verifying How IAM and Lake Formation Behave for the Glue REST Catalog and S3 Tables

#aws #dataengineering

Original Japanese article: Glue REST CatalogとS3TablesのIAMとLake Formationの挙動を確かめる

Introduction

I'm Aki, an AWS Community Builder (@jitepengin).

In my previous article, I hit the Iceberg REST Catalog directly and confirmed the design differences between the Glue endpoint and the S3 Tables endpoint.

In that article I wrote that "accessing through the Glue endpoint requires a Lake Formation grant." This time, I'll deliberately vary the IAM policy and the Lake Formation grant to see exactly how the behavior of each endpoint changes.

Along the way, I'll also check whether GetDataAccess is actually being called via CloudTrail, to peek inside the authorization flow itself.

I previously wrote about the design differences between the Glue and S3 Tables Iceberg REST endpoints in another article, worth a read alongside this one: Hitting the Iceberg REST Catalog Directly: Understanding the Differences Between Glue Data Catalog and S3 Tables

What We're Verifying Today

According to AWS documentation, the Glue REST Catalog is authorized through a combination of IAM policy and Lake Formation grants, while the S3 Tables REST endpoint is authorized through IAM alone.

  • Glue endpoint: IAM authorization → Lake Formation authorization
  • S3 Tables endpoint: s3tables IAM authorization only

Let's verify how this actually plays out, using the following matrix:

Condition Glue endpoint S3 Tables endpoint
IAM ✓ / LF ✓ 200 (baseline) 200 (baseline)
IAM ✓ / LF ✗ ? ?
IAM ✗ / LF ✓ ? ?

The IAM ✓ / LF ✗ cell is especially important:

  • If only the Glue endpoint returns 403 → Glue is genuinely enforcing Lake Formation authorization
  • If only the S3 Tables endpoint returns 200 → that confirms the difference in authorization flow

After each cell, I'll also check whether lakeformation:GetDataAccess was called via CloudTrail, to visualize the internal authorization flow.

Test Environment Setup

I'm reusing the resources created in the previous article.

  • Table bucket: penguin-rest-test
  • Namespace: analytics
  • Table: daily_sales
  • Glue integration: enabled
  • register-resource --with-federation: already configured (vended credentials are issued)

Note

In this test, when using Glue REST Catalog's federation, the Lake Formation permission-evaluation target was the IAM role used to vend credentials. This can be confirmed from the CloudTrail GetDataAccess event, where the session that assumed that IAM role is recorded as the userIdentity.

"IAM ✓" state: the test IAM user has the following policy attached.

{
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["glue:GetCatalog", "glue:GetDatabase", "glue:GetTable"],
      "Resource": "*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "s3tables:GetTableBucket", "s3tables:GetNamespace", "s3tables:GetTable",
        "s3tables:GetTableData", "s3tables:GetTableMetadataLocation"
      ],
      "Resource": [
        "arn:aws:s3tables:ap-northeast-1:123456789012:bucket/penguin-rest-test",
        "arn:aws:s3tables:ap-northeast-1:123456789012:bucket/penguin-rest-test/table/*"
      ]
    },
    {
      "Effect": "Allow",
      "Action": ["lakeformation:GetDataAccess"],
      "Resource": "*"
    }
  ]
}
Enter fullscreen mode Exit fullscreen mode

"LF ✓" state: the IAM role used to vend credentials (penguin-irc-test-role) has the following Lake Formation grants.

  • Database (analytics): DESCRIBE
  • Table (daily_sales): SELECT, DESCRIBE

IAM ✓ / LF ✓ (Baseline State)

Glue Endpoint 

awscurl --service glue --region ap-northeast-1 --profile penguin-irc-test \
  "https://glue.ap-northeast-1.amazonaws.com/iceberg/v1/catalogs/123456789012:s3tablescatalog:penguin-rest-test/namespaces/analytics/tables/daily_sales"
Enter fullscreen mode Exit fullscreen mode

Result (excerpt):

{
  "config": {
    "s3.access-key-id": "ASIAXXXXXXXXXXXXXXXX",
    "s3.secret-access-key": "(masked)",
    "s3.session-token": "(masked)"
  },
  "metadata-location": "s3://...(omitted)"
}
Enter fullscreen mode Exit fullscreen mode

HTTP 200, with a response containing vended credentials.

S3 Tables Endpoint 

BUCKET_ARN_PATH="arn:aws:s3tables:ap-northeast-1:123456789012:bucket%2Fpenguin-rest-test"

awscurl --service s3tables --region ap-northeast-1 --profile penguin-irc-test \
  "https://s3tables.ap-northeast-1.amazonaws.com/iceberg/v1/${BUCKET_ARN_PATH}/namespaces/analytics/tables/daily_sales"
Enter fullscreen mode Exit fullscreen mode

Result (excerpt):

{
  "config": {
    "tableBucketId": "3b2a6702-dd99-44f2-bc03-2f9f6b273104",
    "namespaceId": "20576faa-861e-499f-a30f-4d6c7550dd55",
    "tableId": "34c72c19-610d-4e5c-bee4-6ae9f093c583"
  },
  "metadata-location": "s3://...(omitted)"
}
Enter fullscreen mode Exit fullscreen mode

HTTP 200, with a response containing internal IDs (no credentials).

Checking CloudTrail (Glue Endpoint → GetDataAccess) 

aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=EventName,AttributeValue=GetDataAccess \
  --region ap-northeast-1 --max-results 3
Enter fullscreen mode Exit fullscreen mode

Result (excerpt):

{
  "EventName": "GetDataAccess",
  "EventTime": "2026-07-10T14:39:35+09:00",
  "EventSource": "lakeformation.amazonaws.com",
  "Username": "penguin-irc-test-session",
  "CloudTrailEvent": "...\"invokedBy\":\"glue.amazonaws.com\"..."
}
Enter fullscreen mode Exit fullscreen mode

GetDataAccess is recorded with invokedBy: glue.amazonaws.com. The event was logged within seconds of when the request was sent, which shows that Glue queries Lake Formation synchronously.

No GetDataAccess was recorded for the request made against the S3 Tables endpoint.

Checking CloudTrail (the S3 Tables Endpoint's Own Call)

In addition to confirming that GetDataAccess isn't recorded, let's also check how the S3 Tables endpoint's own API call (GetTableMetadataLocation) shows up in CloudTrail.

aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=EventName,AttributeValue=GetTableMetadataLocation \
  --region ap-northeast-1 --max-results 3 --profile toyano
Enter fullscreen mode Exit fullscreen mode

Result (excerpt, masked):

{
  "EventName": "GetTableMetadataLocation",
  "EventTime": "2026-07-10T14:41:05+09:00",
  "EventSource": "s3tables.amazonaws.com",
  "Username": "penguin-irc-test-session",
  "CloudTrailEvent": "{\"userIdentity\":{\"type\":\"AssumedRole\",\"arn\":\"arn:aws:sts::123456789012:assumed-role/penguin-irc-test-role/penguin-irc-test-session\"},\"eventSource\":\"s3tables.amazonaws.com\",\"eventName\":\"GetTableMetadataLocation\",...}"
}
Enter fullscreen mode Exit fullscreen mode

The eventSource is recorded simply as s3tables.amazonaws.com on its own, and there's no field equivalent to the invokedBy we saw in the Glue-side GetDataAccess event anywhere in the CloudTrailEvent. This confirms, from this angle as well, that authorization on the S3 Tables endpoint is a direct IAM evaluation of the caller's own credentials, rather than a delegated call through another service.

Whether Username shows up as penguin-irc-test (the IAM user itself) or penguin-irc-test-session (the session for the role used to vend credentials) lines up exactly with userIdentity.type (IAMUser vs. AssumedRole), giving us a way to tell which principal actually processed each request.

IAM ✓ / LF ✗ (Removing the LF Grant)

Let's remove the Lake Formation grant from the IAM role used to vend credentials (penguin-irc-test-role).

aws lakeformation revoke-permissions \
  --principal DataLakePrincipalIdentifier=arn:aws:iam::123456789012:role/penguin-irc-test-role \
  --permissions "SELECT" "DESCRIBE" \
  --resource '{"Table":{"CatalogId":"123456789012:s3tablescatalog/penguin-rest-test","DatabaseName":"analytics","Name":"daily_sales"}}'

aws lakeformation revoke-permissions \
  --principal DataLakePrincipalIdentifier=arn:aws:iam::123456789012:role/penguin-irc-test-role \
  --permissions "DESCRIBE" \
  --resource '{"Database":{"CatalogId":"123456789012:s3tablescatalog/penguin-rest-test","Name":"analytics"}}'
Enter fullscreen mode Exit fullscreen mode

Glue Endpoint 

{
  "error": {
    "code": 403,
    "message": "Insufficient Lake Formation permission(s): Required Describe on daily_sales",
    "type": "AccessDeniedException"
  }
}
Enter fullscreen mode Exit fullscreen mode

HTTP 403, with an error indicating insufficient Lake Formation permissions.

S3 Tables Endpoint 

{
  "config": {
    "tableBucketId": "3b2a6702-dd99-44f2-bc03-2f9f6b273104",
    "namespaceId": "20576faa-861e-499f-a30f-4d6c7550dd55",
    "tableId": "34c72c19-610d-4e5c-bee4-6ae9f093c583"
  },
  "metadata-location": "s3://...(omitted)"
}
Enter fullscreen mode Exit fullscreen mode

Still 200, no change. Whether or not the Lake Formation grant exists has no effect on authorization for the S3 Tables endpoint.

Checking CloudTrail (Glue Endpoint → GetDataAccess) 

aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=EventName,AttributeValue=GetDataAccess \
  --region ap-northeast-1 --max-results 3
Enter fullscreen mode Exit fullscreen mode

Result (excerpt):

{
  "EventName": "GetDataAccess",
  "EventTime": "2026-07-10T14:51:29+09:00",
  "EventSource": "lakeformation.amazonaws.com",
  "Username": "penguin-irc-test-session",
  "CloudTrailEvent": "...\"invokedBy\":\"glue.amazonaws.com\"..."
}
Enter fullscreen mode Exit fullscreen mode

Even after removing the Lake Formation grant, GetDataAccess is recorded with invokedBy: glue.amazonaws.com, at roughly the same time the request was sent. For the table-retrieval operation tested here on the Glue endpoint, GetDataAccess is recorded even in the 403 case, confirming that the request does reach Lake Formation's permission evaluation.

IAM ✗ / LF ✓ (Removing the s3tables IAM Actions)

After restoring the Lake Formation grant on the role, let's remove the s3tables-related actions from the IAM policy.

aws iam put-user-policy \
  --user-name penguin-irc-test \
  --policy-name GlueIRCMinimal \
  --policy-document '{
    "Version": "2012-10-17",
    "Statement": [
      {
        "Effect": "Allow",
        "Action": ["glue:GetCatalog","glue:GetDatabase","glue:GetTable"],
        "Resource": "*"
      },
      {
        "Effect": "Allow",
        "Action": ["lakeformation:GetDataAccess"],
        "Resource": "*"
      }
    ]
  }'
Enter fullscreen mode Exit fullscreen mode

Glue Endpoint 

{
  "error": {
    "code": 403,
    "message": "From federation source: User: arn:aws:iam::123456789012:user/penguin-irc-test is not authorized to perform: s3tables:GetTableBucket on resource: arn:aws:s3tables:ap-northeast-1:123456789012:bucket/penguin-rest-test because no identity-based policy allows the s3tables:GetTableBucket action",
    "type": "AccessDeniedException"
  }
}
Enter fullscreen mode Exit fullscreen mode

HTTP 403. In this test, when the calling IAM user lacked s3tables permissions, we got a From federation source error, confirming that the request never even reaches Lake Formation's evaluation.

S3 Tables Endpoint 

{
  "error": {
    "code": 403,
    "message": "User: arn:aws:iam::123456789012:user/penguin-irc-test is not authorized to perform: s3tables:GetTableMetadataLocation on resource: ...",
    "type": "forbidden"
  }
}
Enter fullscreen mode Exit fullscreen mode

Also 403, unsurprising, since the s3tables IAM actions are missing.

Checking CloudTrail (Glue Endpoint → GetDataAccess) 

aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=EventName,AttributeValue=GetDataAccess \
  --region ap-northeast-1 --max-results 3
Enter fullscreen mode Exit fullscreen mode

Result (excerpt):

No matching GetDataAccess event was found.
Enter fullscreen mode Exit fullscreen mode

No GetDataAccess was recorded corresponding to this request. This confirms that when the s3tables-related IAM permissions are removed, Lake Formation is never even queried in the first place, and the request is denied at the IAM authorization stage.

Results Summary

Condition Glue endpoint S3 Tables endpoint GetDataAccess
IAM ✓ / LF ✓ 200 200 Present (14:39:35, recorded under the federation role's session)
IAM ✓ / LF ✗ 403 200 Present (14:51:29, recorded under the federation role's session)
IAM ✗ / LF ✓ 403 403 Absent

Authorization Flow

The results confirm that each endpoint follows a genuinely different flow.

Glue REST Catalog 

Calling IAM user
    |
    | Checks s3tables IAM permissions
    | via the Glue API
    ↓
Glue REST Catalog
    |
    | GetDataAccess
    | (evaluated as the federation IAM role)
    ↓
Lake Formation grant evaluation
    |
    ↓
Vended credentials issued
Enter fullscreen mode Exit fullscreen mode

Here's what this test confirmed:

  1. In this test, when the calling IAM user lacked s3tables IAM permissions, the request was denied at the IAM authorization stage without ever reaching Lake Formation's evaluation (this is where the From federation source error appears).
  2. Once the s3tables IAM permissions are satisfied, Lake Formation permission evaluation is performed with the IAM role used to vend credentials as the principal.
  3. GetDataAccess is recorded in CloudTrail as invokedBy: glue.amazonaws.com (it's recorded even in the 403 case when LF permissions are missing). The request time and the event time are nearly identical, showing that the evaluation happens synchronously.

S3 Tables REST Endpoint 

Calling IAM user
    |
    | s3tables IAM authorization
    ↓
S3 Tables REST endpoint
Enter fullscreen mode Exit fullscreen mode

This test confirmed that the S3 Tables REST endpoint operates purely on an IAM-based authorization model.

Here's what this test confirmed:

  1. Access is controlled by the calling user's IAM policy for s3tables API authorization (no Lake Formation involvement was observed in this test).
  2. Lake Formation-based permission control was not observed on this S3 Tables REST endpoint access path.
  3. GetDataAccess is not recorded in CloudTrail.

Conclusion

In this article, I confirmed the authorization flow for the Glue REST Catalog and the S3 Tables REST endpoint by switching the IAM policy and Lake Formation grants on and off.

To summarize:

  • For the Glue REST Catalog, Lake Formation's GetDataAccess is executed only after the calling IAM user's s3tables-related IAM permissions are satisfied.
  • Lake Formation's permission evaluation is performed with the IAM role used to vend credentials as the principal, not the calling IAM user.
  • Even when a Lake Formation permission shortfall results in a 403, GetDataAccess is still recorded in CloudTrail, confirming that the request does reach Lake Formation's permission evaluation (the request time and event time are nearly identical).
  • On the other hand, when the calling IAM user's s3tables-related IAM permissions are insufficient, GetDataAccess is not recorded, and Lake Formation's permission evaluation never runs.
  • For the S3 Tables REST endpoint, within the scope of this test, GetDataAccess was never recorded and no Lake Formation involvement was observed. The S3 Tables endpoint's own CloudTrail events also have no field equivalent to invokedBy, further confirming that authorization is a direct IAM evaluation.

By combining CloudTrail with the actual API responses, I was able to observe the real authorization flow, something that isn't visible from the documentation's description of "Lake Formation is used" alone.

The CloudTrail logs also back up the difference that the Glue REST Catalog goes through both IAM and Lake Formation authorization, while the S3 Tables REST endpoint is authorized purely through IAM.

To be fair, this all matched the documented behavior, but going through it hands-on still surfaced plenty of small insights and things I learned along the way.

I've found that combining CloudTrail with API responses like this is a genuinely useful technique for understanding the internal behavior of other AWS services too. I'd like to keep running experiments like this to uncover things about AWS's internals that documentation alone doesn't show.

I hope this article is useful to anyone trying to understand the authorization architecture behind the Iceberg REST Catalog on AWS.

Top comments (0)