Move files from Google Cloud Storage to Azure Blob Storage programmatically

Copying files from Google Cloud Storage to Azure Blob Storage programmatically can be achieved by streaming the data directly from cloud storage to blob storage. We will be using google cloud storage and azure storage client libraries for java in this blog.

Streaming allows to transfer the data without requiring the file to be saved first. Saving the entire file in memory and then writing it to the destination can exhaust the memory leading to OutOfMemoryError if the file is large. Instead, we will iterate through and read some bytes at a time, do some processing and clear them.

ReadChannel

ReadChannel, extended from java.nio.channels.ReadableByteChannel, is a reading channel for reading object’s content. Cloud storage client library comes with a reader method which returns ReadChannel to read blob’s content

BlobOutputStream

BlobOutputStream is a stream for writing into azure blob storage. It is extended from java.io.OutputStream

Maven dependencies
<dependency>
<groupId>com.google.cloud</groupId>
<artifactId>google-cloud-storage</artifactId>
<version>1.70.0</version>
</dependency>
<dependency>
<groupId>com.microsoft.azure</groupId>
<artifactId>azure-storage</artifactId>
<version>8.4.0</version>
</dependency>
(keyvault and azure active directory dependencies will be required for retrieving sas token)

Get Storage service object
StorageOptions options = StorageOptions.newBuilder().setProjectId(projectId).setCredentials(GoogleCredentials.fromStream(new FileInputStream(jsonKeyPath))).build();
Storage storage = options.getService();

Get CloudBlockBlob instance
// assuming sas token is retrieved from Azure keyVault
CloudBlockBlob cloudBlob = new CloudBlockBlob(new URI(contentUri + sasToken));

Stream data
try (ReadChannel reader = storage.reader(bucketName, blobPath);
BlobOutputStream outputStream = blob.openOutputStream()) {
ByteBuffer bytes = ByteBuffer.allocate(64 * 1024);
while (reader.read(bytes) > 0) {
bytes.flip();
outputStream.write(bytes.array());
bytes.clear();
}
} catch (Exception e) {
// handle
}
}