DEV Community

Python-T Point
Python-T Point

Posted on • Originally published at pythontpoint.in

🚀 Creating aws s3 bucket policy with python boto3 tutorial

#googlecloud #cloud #devops #python

🔧 Two Approaches – CLI vs. Boto3 for S3 Policies

aws s3 bucket policy python boto3 tutorial

Creating an AWS S3 bucket policy with Python Boto3 involves building a JSON document and calling the put_bucket_policy API. The same API can be invoked from the AWS CLI with aws s3api put-bucket-policy. The CLI workflow requires a manual edit‑and‑run step, while Boto3 can generate the policy in code, store it under version control, and run it automatically from CI pipelines. Both methods result in the same attached policy, but the operational impact differs significantly.

📑 Table of Contents

  • 🔧 Two Approaches – CLI vs. Boto3 for S3 Policies
  • 🛠️ Boto3 Basics — Why They Matter
  • 📄 Policy Structure — How a Bucket Policy Is Formatted
  • 🚀 Applying the Policy — Using Boto3 to Attach
  • 🔎 Verifying the Attachment
  • ⚠️ Common Error – Malformed JSON
  • 🔐 Advanced Controls — Fine‑tuning Permissions
  • 🛡️ Testing Conditional Access
  • 🟩 Final Thoughts
  • ❓ Frequently Asked Questions
  • How do I list the current bucket policy using Boto3?
  • Can I attach a policy to a bucket that already has one?
  • Is there a limit on the size of a bucket policy?
  • 📚 References & Further Reading

🛠️ Boto3 Basics — Why They Matter

A Boto3 client is a thin wrapper that handles request signing (Signature V4), automatic retries, and pagination, allowing direct calls to AWS service endpoints.

# create_policy.py
import boto3
import json # Create a session using default credentials (env vars or ~/.aws/credentials)
session = boto3.Session()
s3 = session.client('s3')
Enter fullscreen mode Exit fullscreen mode

What this does: (More onPythonTPoint tutorials)

  • session = boto3.Session(): loads credentials and region from the environment or shared config files.
  • s3 = session.client( 's3'): creates a low‑level client for the S3 service, exposing methods such as put_bucket_policy.

Why this, not the obvious alternative? Directly using boto3.client avoids loading the higher‑level resource layer, reducing memory footprint for short scripts.

Key point: Boto3 abstracts the AWS signing process, so secret keys never appear in source code.

📄 Policy Structure — How a Bucket Policy Is Formatted

A bucket policy is a JSON document evaluated by S3 for every request to the bucket; the service parses it into an internal policy tree and applies the statements in order.

# bucket_policy.json
{ "Version": "2012-10-17", "Statement": [ { "Sid": "AllowReadOnly", "Effect": "Allow", "Principal": "*", "Action": ["s3:GetObject"], "Resource": ["arn:aws:s3:::example-bucket/*"] } ]
}
Enter fullscreen mode Exit fullscreen mode

What this does:

  • Version: selects the policy language version; AWS requires the 2012‑10‑17 version for all current features.
  • Statement: an array of permission blocks; each block defines a single rule.
  • Sid: optional identifier useful for auditing and debugging.
  • Effect: either Allow or Deny – the engine short‑circuits on the first matching Deny.
  • Principal: the AWS identity (or * for public) to which the rule applies.
  • Action: the S3 operations covered; here only s3:GetObject is permitted.
  • Resource: the ARN pattern that limits the rule to objects under the bucket.

According to the AWS S3 documentation, bucket policies are evaluated after any IAM user policies, giving the bucket owner final control over access.

Aspect Bucket Policy IAM Policy
Scope Applies to a specific bucket Applies to an IAM identity across all services
Evaluation Order Evaluated after IAM policies Evaluated before bucket policies
Typical Use Public read/write, cross‑account access User‑level permissions, role delegation

Key point: Bucket policies provide a “last‑mile” guard that can override broader IAM permissions. (Also read: 🐍 Mastering python classes with dataclasses tutorial for clean code)

🚀 Applying the Policy — Using Boto3 to Attach

The put_bucket_policy method sends the JSON document to S3 over HTTPS; S3 validates the JSON syntax and stores the policy atomically.

# attach_policy.py
import boto3
import json s3 = boto3.client('s3')
bucket_name = 'example-bucket' # Load policy from file
with open('bucket_policy.json', 'r') as f: policy = json.load(f) # Attach the policy
response = s3.put_bucket_policy( Bucket=bucket_name, Policy=json.dumps(policy)
) print('Policy attached, request ID:', response['ResponseMetadata']['RequestId'])
Enter fullscreen mode Exit fullscreen mode

What this does:

  • json.load(f): parses the policy file into a Python dict.
  • s3.put_bucket_policy( …): makes a signed HTTPS request; S3 validates the JSON and persists it.
  • ResponseMetadata.RequestId: uniquely identifies the operation for troubleshooting.

Why this, not the obvious alternative? Calling put_bucket_policy directly from code eliminates a separate CLI step, enabling full automation in deployment scripts.

🔎 Verifying the Attachment

Retrieve the policy to confirm it matches the source.

$ aws s3api get-bucket-policy -bucket example-bucket
{ "Policy": "{\"Version\":\"2012-10-17\",\"Statement\":[{...}]}"
}
Enter fullscreen mode Exit fullscreen mode

The output shows the stored policy as a single JSON string; compare it with the original file using diff or a simple Python assertion. (Also read: ☁️ aws cloudformation vs terraform for python deployments — which one should you use?)

⚠️ Common Error – Malformed JSON

If the JSON is invalid, S3 returns a MalformedPolicy error.

$ python attach_policy.py
Traceback (most recent call last): File "attach_policy.py", line 12, in  s3.put_bucket_policy(Bucket=bucket_name, Policy=json.dumps(policy)) File ".../site-packages/boto3/resources/factory.py", line 124, in wrapper raise e
botocore.exceptions.ClientError: An error occurred (MalformedPolicy) when calling the PutBucketPolicy operation: The policy is not well-formed.
Enter fullscreen mode Exit fullscreen mode

Fixing the syntax (e.g., missing or stray commas) resolves the issue; the API validates the policy before persisting it.

Key point: Immediate verification catches syntax errors before they affect production traffic.

🔐 Advanced Controls — Fine‑tuning Permissions

Adding condition keys to a bucket policy restricts access based on source IP, TLS usage, or VPC endpoint; S3 evaluates these conditions during request authorization.

# bucket_policy_advanced.json
{ "Version": "2012-10-17", "Statement": [ { "Sid": "AllowFromCorporateIP", "Effect": "Allow", "Principal": "*", "Action": ["s3:GetObject"], "Resource": ["arn:aws:s3:::example-bucket/*"], "Condition": { "IpAddress": {"aws:SourceIp": "203.0.113.0/24"}, "Bool": {"aws:SecureTransport": "true"} } } ]
}
Enter fullscreen mode Exit fullscreen mode

What this does:

  • IpAddress condition: limits the request to the specified CIDR block, preventing access from outside the corporate network.
  • Bool condition on SecureTransport: forces HTTPS; any HTTP request is denied before reaching the bucket.

🛡️ Testing Conditional Access

Simulate a request from an allowed IP using the CLI.

$ aws s3api get-object -bucket example-bucket -key test.txt /dev/null -endpoint-url https://s3.amazonaws.com
download: s3://example-bucket/test.txt to ./test.txt
Enter fullscreen mode Exit fullscreen mode

Then test from a disallowed IP (replace with a different network).

$ curl -I https://example-bucket.s3.amazonaws.com/test.txt
HTTP/1.1 403 Forbidden
x-amz-error-code: AccessDenied
Enter fullscreen mode Exit fullscreen mode

The 403 response confirms the condition is enforced.

Key point: Condition keys provide a lightweight, server‑side alternative to application‑level checks.

🟩 Final Thoughts

Automating S3 bucket policy management with Boto3 turns a manual security step into repeatable code. By constructing the policy as JSON, attaching it with put_bucket_policy, and adding condition keys for contextual controls, you gain full programmatic oversight of bucket access. The approach scales with CI/CD pipelines, reduces human error, and aligns with the principle of infrastructure as code.

For environments that enforce least‑privilege access across many buckets, embedding the policy logic in a Python module enables version control, automated testing, and rapid rollout of security updates without leaving the console.

❓ Frequently Asked Questions

How do I list the current bucket policy using Boto3?

Call s3.get_bucket_policy(Bucket='my-bucket'); the response contains the Policy field as a JSON string.

Can I attach a policy to a bucket that already has one?

Yes. put_bucket_policy overwrites the existing policy atomically. Retrieve the current policy first if you need to merge changes.

Is there a limit on the size of a bucket policy?

AWS limits a bucket policy to 20 KB. If you exceed this, consider using multiple statements or consolidating policies.

💡 Want to practise this hands-on? DigitalOcean gives new accounts $200 free credit for 60 days — enough to spin up a full Linux/Docker/Kubernetes environment at no cost.

📚 Recommended reading: Best DevOps & cloud books on Amazon — from Linux fundamentals to Kubernetes in production, curated for working engineers.

📚 References & Further Reading

  • Official AWS S3 bucket policy guide — detailed description of policy elements: aws.amazon.com
  • AWS CLI reference for S3 bucket policies — command‑line equivalents of Boto3 calls: aws.amazon.com

Top comments (0)