DEV Community

irena-sayv
irena-sayv

Posted on • Edited on

Move files from Google Cloud Storage to Azure Blob Storage programmatically

Copying files from Google Cloud Storage to Azure Blob Storage programmatically can be achieved by streaming the data directly from cloud storage to blob storage. We will be using google cloud storage and azure storage client libraries for java in this blog.

Streaming allows to transfer the data without requiring the file to be saved first. Saving the entire file in memory and then writing it to the destination can exhaust the memory leading to OutOfMemoryError if the file is large. Instead, we will iterate through and read some bytes at a time, do some processing and clear them.

ReadChannel

ReadChannel, extended from java.nio.channels.ReadableByteChannel, is a reading channel for reading object’s content. Cloud storage client library comes with a reader method which returns ReadChannel to read blob’s content

BlobOutputStream

BlobOutputStream is a stream for writing into azure blob storage. It is extended from java.io.OutputStream

Maven dependencies
<dependency>
<groupId>com.google.cloud</groupId>
<artifactId>google-cloud-storage</artifactId>
<version>1.70.0</version>
</dependency>
<dependency>
<groupId>com.microsoft.azure</groupId>
<artifactId>azure-storage</artifactId>
<version>8.4.0</version>
</dependency>

(keyvault and azure active directory dependencies will be required for retrieving sas token)

Get Storage service object
StorageOptions options = StorageOptions.newBuilder().setProjectId(projectId).setCredentials(GoogleCredentials.fromStream(new FileInputStream(jsonKeyPath))).build();
Storage storage = options.getService();

Get CloudBlockBlob instance
// assuming sas token is retrieved from Azure keyVault
CloudBlockBlob cloudBlob = new CloudBlockBlob(new URI(contentUri + sasToken));

Stream data
try (ReadChannel reader = storage.reader(bucketName, blobPath);
BlobOutputStream outputStream = blob.openOutputStream()) {
ByteBuffer bytes = ByteBuffer.allocate(64 * 1024);
while (reader.read(bytes) > 0) {
bytes.flip();
outputStream.write(bytes.array());
bytes.clear();
}
} catch (Exception e) {
// handle
}
}

Billboard image

The Next Generation Developer Platform

Coherence is the first Platform-as-a-Service you can control. Unlike "black-box" platforms that are opinionated about the infra you can deploy, Coherence is powered by CNC, the open-source IaC framework, which offers limitless customization.

Learn more

Top comments (0)

A Workflow Copilot. Tailored to You.

Pieces.app image

Our desktop app, with its intelligent copilot, streamlines coding by generating snippets, extracting code from screenshots, and accelerating problem-solving.

Read the docs