DEV Community

AWS Community Builders profile image Aki
Aki for AWS Community Builders

Posted on

Comparing the Permission Models of the Iceberg REST APIs Provided by Glue Data Catalog and S3 Tables

#aws #dataengineering

Original Japanese article: Glue Data CatalogとS3TablesのそれぞれのIceberg REST APIの権限モデルを比べてみる

Introduction

I'm Aki, an AWS Community Builder (@jitepengin).

Across two previous articles, I hit the Iceberg REST Catalog directly and compared the design of the Glue endpoint and the S3 Tables endpoint:

Both of those took the angle of "looking at the same table through two different entry points." This time I wanted to try a different angle: laying the official per-operation permission mapping tables for both endpoints side by side.

For both endpoints, AWS publishes a table mapping each Iceberg REST operation to its IAM action, its Lake Formation permission, and its CloudTrail event name. Just placing these two tables next to each other turned up some interesting findings and design differences, which I've summarized below.

This time, rather than hands-on testing like before, this is a documentation-based comparison. Wherever something lines up with what I actually verified in the previous articles, I've noted that explicitly.

The official documentation I referenced:

Lining Up the Permission Mapping by Operation

Operation Glue: IAM action Glue: LF permission S3 Tables: IAM action
getConfig glue:GetCatalog Not required s3tables:GetTableBucket
listNamespaces glue:GetDatabase ALL, DESCRIBE, SELECT s3tables:ListNamespaces
createNamespace glue:CreateDatabase ALL, DESCRIBE, SELECT s3tables:CreateNamespace
loadNamespaceMetadata glue:GetDatabase ALL, DESCRIBE, SELECT s3tables:GetNamespace
updateNamespaceProperties glue:UpdateDatabase ALL, ALTER (not listed)
dropNamespace glue:DeleteDatabase ALL, DROP s3tables:DeleteNamespace
namespaceExists (not listed) - s3tables:GetNamespace
listTables glue:GetTables ALL, SELECT, DESCRIBE s3tables:ListTables
createTable glue:CreateTable ALL, CREATE_TABLE s3tables:CreateTable + s3tables:PutTableData
loadTable glue:GetTable ALL, SELECT, DESCRIBE s3tables:GetTableMetadataLocation + s3tables:GetTableData
updateTable glue:UpdateTable ALL, ALTER s3tables:UpdateTableMetadataLocation + s3tables:PutTableData + s3tables:GetTableData
dropTable glue:DeleteTable ALL, DROP s3tables:DeleteTable
renameTable (not listed; only possible via the extension API's UpdateTable) - s3tables:RenameTable
tableExists glue:GetTable ALL, SELECT, DESCRIBE s3tables:GetTable

Even just laying these side by side surfaces several differences.

The Number of IAM Actions Differs for Data Operations

The first thing that jumps out is here. For operations that touch data, such as loadTable, createTable, and updateTable, the number of IAM actions required differs between the two.

The Glue endpoint's loadTable needs only one action, glue:GetTable, while the S3 Tables endpoint's loadTable needs two: s3tables:GetTableMetadataLocation and s3tables:GetTableData. createTable and updateTable show the same pattern: the S3 Tables side splits requirements into a "metadata operation action" and a "data body operation action."

This lines up with something I confirmed in a previous article: calling S3 Tables' LoadTable logs a GetTableMetadataLocation management event in CloudTrail.

What this suggests is that S3 Tables cleanly separates access to metadata from access to the data body at the API level, while the Glue endpoint consolidates everything into the single glue:GetTable action and delegates the actual storage access to Lake Formation's vended credentials. That, I think, is the design difference at play here.

Note

The s3tables:CreateTable + s3tables:PutTableData pair shown above is specifically the permission set for the createTable operation as defined in the Iceberg REST API reference (s3-tables-integrating-open-source.html). A separate official document on SQL semantics (https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-tables-sql.html) states that the CreateTable API only creates an empty table inside the table bucket, and that setting the schema requires two more actions, UpdateTableMetadataLocation and GetTableMetadataLocation, for four in total.

In other words, "the API call that creates the table" and "actually getting a usable table with a schema set" require different numbers of permissions. If you only look at the REST API's permission table and assume two actions are enough, you can get tripped up during schema setup.

The renameTable / namespaceExists Difference

The next difference: renameTable appears in S3 Tables' list of supported APIs but not in Glue's.

Glue's official documentation has no independent renameTable operation. Instead, a note on the AWS Glue extension API's (/extensions/v1/...) StartUpdateTableTransaction says: "A rename operation can also be performed through this API. In that case, the caller must also have glue:CreateTable or the Lake Formation CREATE_TABLE permission for the destination table." So on the Glue side, renaming isn't its own operation; it's treated as a kind of update transaction.

namespaceExists, meanwhile, exists as its own operation on the S3 Tables side (HEAD /v1/{prefix}/namespaces/{namespace}), but I couldn't find it anywhere in Glue's list of standard Iceberg REST Catalog API operations. Even though both implement the same Iceberg REST spec, the design decision of "what gets carved out as its own API" clearly differs.

The updateNamespaceProperties Difference

updateNamespaceProperties existing only on the Glue side is another difference worth calling out. Glue has a clearly defined updateNamespaceProperties operation with the IAM action glue:UpdateDatabase, but this operation doesn't exist at all in S3 Tables' list of operations. S3 Tables' documented limitations say "only the owner property is supported for namespaces," but there's no corresponding independent operation listed in the official operations table.

This might get added in a future update, but as things stand, it's another point of difference between the two.

The dropTable purge Behavior Difference

According to the official documentation, S3 Tables' dropTable returns a 400 Bad Request if you specify purge=false, meaning purge must be true. Glue's DeleteTable (for tables on S3), on the other hand, is documented as failing the operation and not deleting the data if you specify purge=true.

Same "delete the table" operation, but the required value of purge looks like it's flipped between the two according to the docs.

This looked like a likely source of mistakes, so I decided to actually check it hands-on. Testing it turned up something interesting.

I tried it first against the S3 Tables endpoint.

S3 Tables' purge Behavior

I created a table, then called dropTable with no parameters at all:

python -m awscurl --service s3tables --region ap-northeast-1 --profile your-profile \
  -X DELETE \
  "https://s3tables.ap-northeast-1.amazonaws.com/iceberg/v1/${BUCKET_ARN_PATH}/namespaces/analytics/tables/purge_test"
Enter fullscreen mode Exit fullscreen mode

Result (excerpt):

{
  "error": {
    "code": 400,
    "message": "DropTable operation failed. S3 Tables only supports dropping tables with purge enabled.",
    "type": "purge_enabled"
  }
}
Enter fullscreen mode Exit fullscreen mode

As documented, this returned 400, and the table stayed in the listing.

Next, I re-ran it with the query parameter purge=true and got the exact same error. It turns out that, per the Iceberg REST Catalog spec, the query parameter for specifying purge on dropTable isn't purge; it's purgeRequested.

python -m awscurl --service s3tables --region ap-northeast-1 --profile your-profile \
  -X DELETE \
  "https://s3tables.ap-northeast-1.amazonaws.com/iceberg/v1/${BUCKET_ARN_PATH}/namespaces/analytics/tables/purge_test?purgeRequested=true"
Enter fullscreen mode Exit fullscreen mode

This one succeeded with no response body, and I confirmed via the listing that the table had actually been deleted.

Note

The gist of "purge must be true" was correct per the documentation, but assuming the query parameter is named purge will trip you up. It's actually purgeRequested.

If you use both endpoints together, this alone is fertile ground for implementation mistakes.

Glue Data Catalog's purge Behavior

I then ran the same test on the Glue side.

My test environment has the Glue endpoint reaching S3 Tables-backed tables through the s3tablescatalog federated catalog. So I first called DeleteTable against this table via the Glue endpoint with purgeRequested=true specified, and it succeeded without any fuss.

python -m awscurl --service glue --region ap-northeast-1 --profile your-profile \
  -X DELETE \
  "https://glue.ap-northeast-1.amazonaws.com/iceberg/v1/catalogs/123456789012:s3tablescatalog:penguin-rest-test/namespaces/analytics/tables/purge_test_glue?purgeRequested=true"
Enter fullscreen mode Exit fullscreen mode

This contradicts the official documentation's claim that "Glue's DeleteTable fails if purge=true." So I set up another table and tried deleting it via Glue without specifying purgeRequested at all.

python -m awscurl --service glue --region ap-northeast-1 --profile your-profile \
  -X DELETE \
  "https://glue.ap-northeast-1.amazonaws.com/iceberg/v1/catalogs/123456789012:s3tablescatalog:penguin-rest-test/namespaces/analytics/tables/purge_test_glue2"
Enter fullscreen mode Exit fullscreen mode

Result (excerpt):

{
  "error": {
    "code": 400,
    "message": "PurgeRequested must be true for S3 federated iceberg tables.",
    "type": "InvalidInputException"
  }
}
Enter fullscreen mode Exit fullscreen mode

The error message spells it out clearly: "S3 federated iceberg tables." So the Glue endpoint internally distinguishes whether a table's underlying entity is S3 Tables-backed (federated), and in that case it demands purgeRequested=true.

This tells us that the purge behavior of DeleteTable isn't determined by which endpoint you're hitting (Glue or S3 Tables). What determines it is the underlying entity of the table: whether the backend is an S3 Tables-federated table or a plain table on a general-purpose S3 bucket.

The documentation's statement that "Glue's DeleteTable fails for S3-backed tables if purge=true" is presumably describing a genuinely Glue-native Iceberg table on a general-purpose S3 bucket. For a table backed by S3 Tables, even when accessed through Glue, S3 Tables' own constraint (purgeRequested=true required) appears to pass straight through.

To test this hypothesis, I created an Iceberg table on a general-purpose S3 bucket via Athena and ran the same test through the Glue endpoint.

First, deleting without specifying purgeRequested succeeded:

python -m awscurl --service glue --region ap-northeast-1 --profile your-profile \
  -X DELETE \
  "https://glue.ap-northeast-1.amazonaws.com/iceberg/v1/catalogs/123456789012/namespaces/general_iceberg_db/tables/purge_test_general"
Enter fullscreen mode Exit fullscreen mode

Then, on a second table, specifying purgeRequested=true failed:

python -m awscurl --service glue --region ap-northeast-1 --profile your-profile \
  -X DELETE \
  "https://glue.ap-northeast-1.amazonaws.com/iceberg/v1/catalogs/123456789012/namespaces/general_iceberg_db/tables/purge_test_general2?purgeRequested=true"
Enter fullscreen mode Exit fullscreen mode

Result (excerpt):

{
  "error": {
    "code": 400,
    "message": "PurgeRequested cannot be true for Glue iceberg tables.",
    "type": "InvalidInputException"
  }
}
Enter fullscreen mode Exit fullscreen mode

This mirrors the earlier error for the S3 Tables-backed table (PurgeRequested must be true for S3 federated iceberg tables.) almost perfectly. AWS itself clearly distinguishes "Glue iceberg tables" from "S3 federated iceberg tables" in its code, and imposes opposite purgeRequested requirements on each.

Summary of purge Behavior for Each

Here's the full result across all four cases tested:

Table entity Without purgeRequested With purgeRequested=true
S3 Tables-backed (via Glue endpoint) 400 "PurgeRequested must be true for S3 federated iceberg tables" Success
S3 Tables-backed (direct S3 Tables endpoint) 400 "S3 Tables only supports dropping tables with purge enabled" Success
General-purpose S3, Glue-native table (via Glue endpoint) Success 400 "PurgeRequested cannot be true for Glue iceberg tables"

Both failure cases without purgeRequested return the same status code, 400. The nature of the error is consistent regardless of which endpoint you hit; the only difference is what the message says caused the failure.

It's not the entry point (Glue endpoint vs. S3 Tables endpoint) that decides the purgeRequested requirement; it's the underlying table entity (Glue Iceberg Tables vs. S3 Federated Iceberg Tables). Both the hands-on results and the error messages back this up.

I'd honestly prefer they picked one convention and stuck with it...

CloudTrail Granularity Also Reflects the API Design Split

S3 Tables' official documentation states clearly that "LoadTable generates a GetTableMetadataLocation management event and a GetTableData data event." This lines up with the IAM action split described above. S3 Tables appears to be designed to separate "metadata" from "the data body" at the audit-log level too.

On the Glue side, the official documentation shows each operation's CloudTrail event name matching its IAM action name (glue:GetTable logs as glue:GetTable), with no split like the one seen on S3 Tables. That said, as I confirmed hands-on in a previous article, when the Glue endpoint calls Lake Formation's GetDataAccess, that gets logged separately as its own event. It seems Glue's audit trail is split across two distinct lineages: "the log of the API operation itself" and "the Lake Formation authorization log."

Conclusion

This time, taking a different approach from the hands-on investigations I've done so far, I laid the official documentation-based permission mapping tables for Glue Data Catalog's and S3 Tables' respective Iceberg REST APIs side by side.

To summarize:

  • Granularity of data-operation permissions: Glue consolidates each operation into a single IAM action, while S3 Tables splits metadata and data-body actions apart (createTable needs only two actions as a REST operation, but per the SQL semantics documentation, actually getting a usable table with a schema set requires four).
  • Rename operations: Glue has no independent API for this; it's handled as part of an update transaction. S3 Tables has its own dedicated renameTable operation.
  • namespaceExists: exists on the S3 Tables side but not in Glue's standard API list.
  • updateNamespaceProperties: exists on the Glue side but has no corresponding operation on S3 Tables (only the owner property is supported, in a limited way).
  • dropTable's purge behavior: the query parameter is purgeRequested, not purge. Whether purge is required turns out to be determined not by "Glue endpoint vs. S3 Tables endpoint" but by the table's underlying entity (S3 Tables-federated vs. Glue-native on general-purpose S3), and the requirement is the exact opposite in each case, which I confirmed hands-on across all four scenarios.
  • CloudTrail granularity: S3 Tables logs separate events for metadata operations and data operations, while Glue logs one event per operation, with Lake Formation evaluation showing up as a separate lineage.

Even though both implement the same Iceberg REST Catalog spec, it turns out the granularity of the permission model and the design decisions around what gets carved out as an independent API differ quite a bit between Glue and S3 Tables. I think this is the same structural difference I touched on in earlier articles, Glue as a catalog hub and S3 Tables as a thin translation layer over storage, showing up consistently even at the level of permission mapping tables.

Comparing hands-on results against the documentation turns up things you'd miss looking at either one alone, so I'd like to keep doing this kind of exercise going forward.

I hope this article is useful to anyone choosing between the two endpoints.

Top comments (1)

Collapse
 
valentin_monteiro profile image
Valentin Monteiro •

The purgeRequested finding is the one that'll actually bite people: any wrapper that sets the flag per-endpoint instead of per-table-entity works fine until someone federates a bucket underneath it. Both failure modes come back as 400, so a client can't safely retry with the flag flipped without parsing the message string. Did you find anything in the loadTable response that lets you tell a Glue-native table from an S3-federated one before you attempt the drop?