The new Azure SDK for Java
This blog post uses the upcoming Azure SDK for Java: at the time of this writing, this is still a preview release, but many people have already started using it.
To be more precise, we are talking here of the Azure SDK for Java (August 2019 Preview), also seen as "1.0.0-preview.2" on Maven Central.
This new release is important as it uses some new, modern API guidelines, and also because its asynchronous features are powered by Spring Reactor.
Those new reactive APIs are interesting as they deliver better scalability, and work very well with Spring. Unfortunately, they come at the cost of using an API that is more complex to understand, and more complex to debug: this is why we are doing this blog post!
The problem with uploading data
Uploading data takes time, and usually need a good connection: if you're working with mobile devices, this can definitely be an issue! If we use a thread-based model, this means that sending several files is going to block several threads, and is not going to be very scalable. Or you could put all files in a queue that you would manage yourself: this is probably quite complex to code, and this will prevent you from uploading those files in parallel, so performance won't be very good. This is where Spring Reactor comes into play!
The idea here is to upload that data in an asynchronous, non-blocking way, so we can upload many files in parallel without blocking the system.
Spring Reactor and Spring Webflux
Please note that the example here works with Spring Webflux: this is the reactive version of Spring, and as this works in a totally different way than the classical Spring MVC (which is thread-based), this code won't work with Spring MVC. This is one of the issues when doing reactive programming: it's not really possible to mix reactive code and thread-based code, so all your project will need be made and thought with a reactive API. In my opinion, this is best suited in a microservice architecture, where you will code some specific services with a reactive API, and some with a traditional thread-based API. Reactive microservices will probably be a bit more complex to develop and debug, but will provide better scalability, start-up time, memory consumption, so they will be used for some specific, resource-intensive tasks.
Creating a Storage Account
In the the Azure portal, create a new storage account:
Once it is created, go to that storage account and select "Shared access signature", or SAS. A SAS is a signature that allows to access some resources, for a limited period of time: it is perfect for uploading data on a specific location, without compromising security.
After clicking on "Generate SAS and connection string", copy the last generated text field, named "Blob service SAS URL". This is the one we will use with the Azure SDK for Java.
Add the Azure SDK for Java to the pom.xml
As the Azure SDK for Java preview is on Maven Central, this is just a matter of adding the dependency to the project's pom.xml:
<dependency>
<groupId>com.azure</groupId>
<artifactId>azure-storage-blob</artifactId>
<version>12.0.0-preview.2</version>
</dependency>
Using the new asynchronous API
Let's first have a look at the final code, which is available on https://github.com/jdubois/jhipster-azure-blob-storage/blob/master/src/main/java/com/example/demo/PictureResource.java:
@RestController
@RequestMapping("/api")
public class PictureResource {
Logger log = LoggerFactory.getLogger(PictureResource.class);
@Value("${AZURE_BLOB_ENDPOINT}")
private String endpoint;
@PostMapping("/picture")
public void uploadPicture() throws IOException {
log.debug("Configuring storage client");
BlobServiceAsyncClient client = new BlobServiceClientBuilder()
.endpoint(endpoint)
.buildAsyncClient();
client.createContainer("pictures")
.doOnError((e) -> {
log.info("Container already exists");
})
.flatMap(
(clientResponse) -> {
log.info("Uploading picture");
return clientResponse
.value()
.getBlockBlobAsyncClient("picture.png")
.uploadFromFile("src/main/resources/image.png");
})
.subscribe();
}
}
WARNING This API only works when using Spring Reactive, so please note you need to use this in a Spring Webflux project, and not a Spring MVC project.
Authentication is done using the "Blob service SAS URL" that we copied above, and which is provided using the AZURE_BLOB_ENDPOINT environment variable in this example: please note that the SAS is included in the URL, so there is no need to authenticate elsewhere (there is a credentials() method in the API, that might be misleading, but which is useless in our case). This URL should thus be stored securely, and not commited with your code.
Sending the image uses the Spring Reactor API:
- We create a specific
picturescontainer to store some data - We then use Spring Reactor's API to upload a picture
- And we finish by using the
subscribe()method, which makes our code run asynchronously
As a result, this method will return very quickly, and then the container will be created and the image uploaded, in another thread. This can make debugging harder, but allows our application to accept many more requests, and process them asynchronously.
Improving the reactive code
This tip was provided by Christophe Bornet, many thanks to him!
The previous code is what we usually see in projects, but that can be improved, in order to let Spring Webflux handle the .subscribe() part: this will preserve the backpressure between Spring Webflux and the Azure SDK. Also, that means that errors will be managed by Spring Webflux, instead of being lost: in the case of an error while uploading a file using the Azure SDK, with this new code you will have a clean 500 error.
The change can be seen in this commit, where we replace the .subscribe() by a .then() and we return a Mono<Void> instead of not returning anything. It will be Spring Webflux's responsibility to handle that Mono and call .subscribe().
The resulting code is the following:
@PostMapping("/picture")
public Mono<Void> uploadPicture() throws IOException {
log.debug("Configuring storage client");
BlobServiceAsyncClient client = new BlobServiceClientBuilder()
.endpoint(endpoint)
.buildAsyncClient();
return client.createContainer("pictures")
.doOnError((e) -> {
log.info("Container already exists");
})
.flatMap(
(clientResponse) -> {
log.info("Uploading picture");
return clientResponse
.value()
.getBlockBlobAsyncClient("picture.png")
.uploadFromFile("src/main/resources/image.png");
})
.then();
}
It's a bit more advanced usage of the reactive APIs, but the result should be worth the trouble.
Conclusion and feedback
We are currently lacking documentation and samples on this new asynchronous API in Azure SDK for Java. I believe that it is very important in some specific scenarios like the one we have here, as typically you should not upload or download data in the current thread if you want a scalable application.
This SDK is still in preview, so if you have feedback on this API, please comment on this post!
For example, the current API allows you to create a container (and this will fail if a container already exist) or get an existing container (and this will fail if it does not exist yet). Do you think there should be an option to have something like getOrCreateContainer("name"), that will automatically create a container if it is requested?
Top comments (12)
when I tried executing the example with Spring Boot 2.1.8.RELEASE (webflux starter) got below exception. is Azure SDK required minimum reactor-netty verion ? Spring Boot uses different reactor-netter version. I could see only dependency version difference between both.
reactor.core.Exceptions$ErrorCallbackNotImplemented: java.lang.IllegalStateException: block()/blockFirst()/blockLast() are blocking, which is not supported in thread reactor-http-nio-3
Caused by: java.lang.IllegalStateException: block()/blockFirst()/blockLast() are blocking, which is not supported in thread reactor-http-nio-3
at reactor.core.publisher.BlockingSingleSubscriber.blockingGet(BlockingSingleSubscriber.java:77) ~[reactor-core-3.2.12.RELEASE.jar:3.2.12.RELEASE]
at reactor.core.publisher.Mono.block(Mono.java:1494) ~[reactor-core-3.2.12.RELEASE.jar:3.2.12.RELEASE]
at com.azure.core.implementation.RestProxy.createResponse(RestProxy.java:456) ~[azure-core-1.0.0-preview.3.jar:na]
at com.azure.core.implementation.RestProxy.lambda$handleRestResponseReturnType$5(RestProxy.java:396) ~[azure-core-1.0.0-preview.3.jar:na]
at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:107) ~[reactor-core-3.2.12.RELEASE.jar:3.2.12.RELEASE]
I don't think this issue is linked to the Netty version - it seems you are doing a blocking call at some point, and that's more likely to be the issue.
Did you change anything in the sample code, or just run it without any modification? I might have done an error when copy/pasting the code at some point.
If you have doubts about the Netty versions, I would use the one from Spring Boot, as it's using it far more extensively so you'll reduce the number of potential issues.
I haven't modified the code, only I changed boot version to 2.1.8. But it worked as expected on 2.2.0 milestone and snapshot build.
This is a really important side note. Uploading (or downloading) files is a good use case for a reactive server. Also long living connections like streaming are. Other use cases probably aren’t.
Thanks, this could come in handy. Do you have any suggestions for API built with Micronaut on how to deploy to Azure or better yet is it as supported as Spring?
I'm guessing you only want to deploy a Micronaut application to Azure, so for this you don't really need to learn about the SDK (the SDK is if you want to use some specific Azure service from your Micronaut application - I guess it works the same as any other Java application, but you won't benefit from Spring Webflux like I do in this post).
In that case, we have a tutorial just for you : guides.micronaut.io/micronaut-azur... :-)
Thank you, I've seen the docs before but I was worried about this "JDK 8 installed with JAVA_HOME configured appropriately (cannot be Java 9 or later)". So I got the impresion that Spring has a bit more to offer. There's too much stuff to do with deploying Java to Azure when I was hoping for Azure DevOps to provide simple solution as for other technologies. I guess I will try to set this up on DevOps later whitout Azure CLI and such and I'll hope it will support JDK 11 otherwise it will be switch from Azure or to .NET Core.
Where did you see this Java 8 requirement? I'm using Java 11 for all this sample code, and had no problem at all.
In the link you provided, I saw the link couple of days ago but didn't read it fully just saw the limitation but it's for the computer being used to develop as far as I can see. I'm not sure is it because of other steps the in the documentation or something else.
Wait wait wait... I'm genuinely curious now. Do we really have "React" in JavaScript and "Reactor" in Java now!?
Of yes, you have "reactive" applications in Java, and that's not the same thing as "React" in the front-end! It also doesn't mean what UX/UI people call "reactive".
some alternative for getOrCreateContainer("name")?