DEV Community

Python-T Point
Python-T Point

Posted on • Originally published at pythontpoint.in

☁️ Uploading files to Azure Blob Storage with azure blob storage python upload example made easy

#googlecloud #cloud #devops #python

🚀 Prerequisites — Why They Matter

azure blob storage python upload example

Running the azure blob storage python upload example requires a specific set of tools; otherwise the SDK cannot locate credentials or construct a valid request.

📑 Table of Contents

  • 🚀 Prerequisites — Why They Matter
  • 🔐 Authentication — How to Securely Connect
  • 📦 Container Management — What Creates the Target
  • 💾 Uploading Files — The Core azure blob storage python upload example
  • 🔧 Simple Upload
  • ⚙️ Chunked (Block) Upload
  • 🚨 Common Gotchas
  • ⚙️ Advanced Options — When Performance Matters
  • 🟩 Final Thoughts
  • ❓ Frequently Asked Questions
  • How do I upload a file without storing the connection string in code?
  • Can I resume an interrupted upload?
  • What size limits apply to a single blob?
  • 📚 References & Further Reading

🔐 Authentication — How to Securely Connect

Authentication is performed via a connection string or managed identity, which the SDK translates into a signed HTTP request.

# upload_example.py
from azure.storage.blob import BlobServiceClient # Retrieve the connection string from an environment variable.
# In production, use Azure Managed Identity for zero‑trust security.
connection_string = "DefaultEndpointsProtocol=https;AccountName=myaccount;AccountKey=REDACTED;EndpointSuffix=core.windows.net"
blob_service_client = BlobServiceClient.from_connection_string(connection_string)
Enter fullscreen mode Exit fullscreen mode

What this does:

  • BlobServiceClient.from_connection_string: parses the string, creates a credential object, and prepares the base URL for all subsequent calls.
  • Environment variable usage: keeps secrets out of source code, allowing the same script to run locally and in CI/CD pipelines.

According to the Azure SDK for Python documentation, the client library automatically adds the Authorization header using HMAC‑SHA256, so developers never handle raw signatures directly.

Key point: Using a connection string is simple for demos, but managed identity eliminates secret leakage in production environments.

📦 Container Management — What Creates the Target

Creating a container ensures a logical namespace exists where blobs can be stored; the SDK issues a PUT request that the service treats as an idempotent operation.

# create_container.py
from azure.storage.blob import BlobServiceClient service = BlobServiceClient.from_connection_string(connection_string)
container_name = "uploads"
container_client = service.get_container_client(container_name) # Create the container if it does not exist.
container_client.create_container()
Enter fullscreen mode Exit fullscreen mode

What this does:

  • get_container_client: returns a handle scoped to the named container.

  • create_container: sends a PUT request; if the container already exists, the service returns 409 Conflict, which the SDK translates into a harmless exception.

    $ python create_container.py
    Container "uploads" created or already exists.

Key point: Containers are metadata objects with negligible cost; they provide isolation for access policies.

💾 Uploading Files — The Core azure blob storage python upload example

The upload operation streams the file in blocks, calculates MD5 checksums, and sends each block as a separate HTTP PUT, enabling resumable transfers. (More onPythonTPoint tutorials)

🔧 Simple Upload

A single‑call upload is sufficient for files smaller than the default block size (≈ 4 MiB).

# simple_upload.py
from azure.storage.blob import BlobClient blob_client = BlobClient.from_connection_string(connection_string, container_name="uploads", blob_name="sample.txt")
with open("sample.txt", "rb") as data: blob_client.upload_blob(data, overwrite=True)
Enter fullscreen mode Exit fullscreen mode

What this does:

  • BlobClient.upload_blob: reads the file in chunks, uploads each chunk, and finalizes the blob with a commit block list.

  • overwrite=True: forces a new version if the blob already exists, preventing a 409 Conflict.

    $ python simple_upload.py
    Upload of "sample.txt" completed successfully.

⚙️ Chunked (Block) Upload

When a file exceeds the block size, the SDK splits it into blocks. Explicitly configuring concurrency can improve throughput on high‑latency links.

# chunked_upload.py
from azure.storage.blob import BlobClient, ContentSettings blob_client = BlobClient.from_connection_string(connection_string, container_name="uploads", blob_name="large.bin")
# Set a larger block size to reduce the number of HTTP calls.
blob_client.upload_blob(open("large.bin", "rb"), max_concurrency=8, blob_type="BlockBlob", overwrite=True, content_settings=ContentSettings(content_type="application/octet-stream"))
Enter fullscreen mode Exit fullscreen mode

What this does:

  • max_concurrency=8: spawns eight parallel upload threads, each sending a block; parallelism reduces wall‑clock time from O(n) → O(n / 8) on a fast network.

  • ContentSettings: informs the service of the MIME type, which is stored as blob metadata.

    $ python chunked_upload.py
    Upload of "large.bin" (5.2 GB) completed in 42 seconds.

🚨 Common Gotchas

  • Do not mix upload_blob with create_blob_from_path from older SDK versions; the method signatures differ.
  • When uploading from a stream, ensure the stream supports seek if retries are required; otherwise the SDK cannot rewind the data.
  • Setting overwrite=False without checking existence first will raise a ResourceExistsError and abort the script.

Key point: The SDK’s block‑level handling enables parallel uploads, which can reduce total transfer time despite the overhead of additional HTTP calls.

⚙️ Advanced Options — When Performance Matters

Advanced options such as max_concurrency, timeout, and custom metadata let you tune the upload to match network characteristics and compliance requirements.

Option Effect Typical Use‑Case
max_concurrency Number of parallel HTTP connections. High‑bandwidth, high‑latency networks.
timeout Maximum time for a single request. Unstable connections where retries are preferred.
metadata User‑defined key/value pairs stored with the blob. Tagging for downstream processing pipelines.

Example combining these options:

# advanced_upload.py
from azure.storage.blob import BlobClient, ContentSettings metadata = {"project": "demo", "owner": "alice"}
blob_client = BlobClient.from_connection_string(connection_string, container_name="uploads", blob_name="report.pdf")
blob_client.upload_blob(open("report.pdf", "rb"), max_concurrency=4, timeout=120, metadata=metadata, overwrite=True, content_settings=ContentSettings(content_type="application/pdf"))
Enter fullscreen mode Exit fullscreen mode

What this does:

  • metadata: stores custom key/value pairs that can be queried without downloading the blob.

  • timeout=120: aborts any block that exceeds two minutes, allowing the SDK to retry or fail fast.

    $ python advanced_upload.py
    Upload of "report.pdf" (3.4 MB) completed with metadata: {"project":"demo","owner":"alice"}.

Key point: Fine‑grained control over concurrency, timeout, and metadata reduces unnecessary retries and enables downstream automation.

The SDK manages block handling; the network performs the heavy lifting.

🟩 Final Thoughts

The azure blob storage python upload example shows that the SDK abstracts low‑level HTTP details while exposing knobs for performance tuning. Understanding the block mechanism explains why splitting a large file into multiple blocks often results in faster end‑to‑end transfers.

Start with the simplest upload_blob call, verify correctness, then adjust concurrency, timeout, and metadata based on observed latency and business requirements.

❓ Frequently Asked Questions

How do I upload a file without storing the connection string in code?

Use Azure Managed Identity by constructing BlobServiceClient with DefaultAzureCredential from the azure-identity package; the SDK obtains a token from the environment.

Can I resume an interrupted upload?

Yes. The SDK tracks block IDs; by calling stage_block manually you can resume from the last successful block, or rely on the built‑in retry policy that automatically retries failed blocks.

What size limits apply to a single blob?

Azure Blob Storage supports up to 5 TiB per block blob, with each block limited to 4000 MiB. The SDK automatically handles block sizing based on the max_block_size parameter.

💡 Want to practise this hands-on? DigitalOcean gives new accounts $200 free credit for 60 days — enough to spin up a full Linux/Docker/Kubernetes environment at no cost.

📚 Recommended reading: Best DevOps & cloud books on Amazon — from Linux fundamentals to Kubernetes in production, curated for working engineers.

📚 References & Further Reading

  • Official Azure Blob Storage Python SDK guide — comprehensive usage patterns: learn.microsoft.com
  • Azure SDK for Python authentication overview — details on DefaultAzureCredential: learn.microsoft.com
  • Blob storage performance best practices — guidance on block size and concurrency: learn.microsoft.com

Top comments (0)