DEV Community: Eugen Podaru

Copying blob containers between storage accounts using Azure CLI

Eugen Podaru — Sun, 29 Nov 2020 22:43:10 +0000

TL;DR

TL

In the past few weeks, we have been doing a migration of our platform from one Azure account to another and I have been using this as an opportunity to update different parts of our deployment pipelines. While doing this I have become more familiarized with Azure CLI and soon enough I have become a fan (not a literal fan, just a fan of Azure CLI).

Previously we were using ARM templates to deploy our resource groups and the resources in them. This worked fine, but I did not particularly like them. One of the reasons is because they are way to verbose. Even if you want a resource with all the settings set to their defaults, you still need to define a boatload of JSON. Of course, you do not have to write it all from scratch. You can start from a quick start template, and change only what you need, but it still takes too much effort to mentally parse it all and understand it. Anyway, it is so much nicer, shorter, and easier to read and write using Azure CLI.

Migrating the platform also means migrating the data, not just the resources. As it happens, we have data across a wide range of services: Storage Accounts (in Blobs, Table Storage, File Shares and Data Lake Gen 2), Data Lake Gen 1 and Azure SQL.

A lot of the data migration tasks can be accomplished by using Azure Storage Explorer, such as copying whole blob containers, tables and file shares between accounts, and copying entire file systems between data lake accounts (both Gen 1 and Gen 2). It uses AzCopy underneath, which uses server to server APIs to copy data directly between accounts, so it is rather efficient.

What cannot be done using Azure Storage Explorer is automating any of the above tasks. Also, while copying the blob containers, tables and file shares, you need to do it one by one, which is not great if you have a lot of them in a particular storage account.

As you have probably guessed from the title, you can use Azure CLI to copy all blob containers from one account to the other. There is not much to it, it is just a simple command, currently in preview, that I found while looking for solutions to do it:

Keep in mind that it is currently in preview (Azure CLI v2.15.1), so it might change in future releases. It also utilizes server to server APIs to perform the operation. You can use SAS tokens instead of the connection strings:

You can get the account connection string or generate a SAS token also using Azure CLI:

The nice thing about Azure CLI is that you can include the scripts in your azure pipelines and run them along with other DevOps tasks manually or on some triggers. Bellow is an example of a pipeline that runs every day to copy all blob containers between two storage accounts. You could use it to keep the production and acceptance environments in sync for example:

The pipeline assumes that you have some pipeline variables that define the account names and connection strings. The Subscription variable refers to the Azure service connection to use for this script.

Conveniently, you could use Azure CLI to create both the service connection and the pipeline that runs the Azure CLI task. Mind blown 🤯! You would do that using the Azure DevOps CLI extension. Below are the commands and I leave it up to you to figure out how to use them:

When talking about Azure Storage Explorer I said that underneath it uses AzCopy. Well, AzCopy can also be used to copy all the blob containers between accounts, as well as other interesting scenarios. But that is a matter for a future post!

Dynamic Http Proxy with Azure Functions

Eugen Podaru — Sun, 15 Nov 2020 15:04:02 +0000

In the following article I am going to show you how to build a dynamic http proxy using Azure Functions and truly little code. If you feel like reading code rather than reading an article about code, go straight to GitHub and have it your way.

You might be asking yourself what is a dynamic http proxy and why would I need one? You could think of it as a service registry that also allows you to proxy the http calls to the services it registers, so a dynamic http proxy, DHP or DHTTPP in short.

One possible use case for such a service is between your frontends and your backends. Regardless of how fragmented or dynamic your backend services landscape is, the DHP makes it possible for you to define a single, predictable API surface for the frontends to consume. The frontends also become simpler, by only having to know about one endpoint. You could accomplish the same using a solution like API Management of course, but while for each modified or added backend you need to reconfigure the API Management, the DHP reconfigures itself.

If you used Azure Functions before, you might have stumbled upon Azure Functions Proxies. They are somewhat like API Management and can be used to achieve the same goal of a single API surface for all your services. But just like API Management, they are not dynamic, you would have to add any additional route and potentially redeploy the functions app. I will not be using them for the DHP.

The idea is simple, every discoverable service registers itself with the DHP. After registering, the service will be accessible through the proxy.

Let us start with the service registration. When registering itself a service needs to provide a name, a version and the host where it can be found. I guess a name and a host would have sufficed, but it is nice to have API versioning out-of-the-box and it adds little in terms of extra code to the final solution. In the following code listing you can see how the service registration function looks like:

The function is as simple as it gets. The DHP accepts service registrations at /api/register and the payload for the registrations needs to be a ServiceEntry delivered in the body of the post method. Once it gets the ServiceEntry, it simply saves it to a table in an Azure Table Storage. I will leave service deregistration's (delete) and service queries (get) up to you, since they are not mandatory for the DHP.

You might have noticed that before saving to the table, we set the RegisteredAt property of the ServiceEntry to the current datetime in UTC. This can be useful in multiple ways. If you make your services register only once, then this will only tell you when that happened of course. However, if you make your services register once every few minutes, then suddenly this tells you which services are up, and which are down. Nice and simple!

In the following code listing you can see the ServiceEntry class:

As you can see, I am using the service name and version as partition and row keys, which feels very natural, doesn’t it? Without the service version we would have had to make one of the keys up, which would have been a waste.

This is all for service registration, let us see how the proxy functions look like. It is only one function and you can see it in the following code listing:

It is a simple HTTP triggered function, that accepts all HTTP verbs. Using Azure Functions binding expressions, it maps the partition and row keys of the table input binding to the name and version variables extracted from the route. This way, the runtime does the job of retrieving the matched ServiceEntry for us. Next, it builds the target endpoint using the service host and the rest of the path extracted from the route, adjusts the HttpRequestMessage and sends it using the injected IProxyService. You can see the implementation of the IProxyService in the next code listing:

I bet you thought this will be complicated. Here we are just wrapping the good old HttpClient to make the call to our proxied service. The only thing worth mentioning here is the use of HttpCompletitionOption.ResponseHeadersRead which means the operation should complete as soon as a response is available and headers are read. Since we are proxying the response, there is no need to wait until all the content is read.

As promised, little code, yet quite capable!

Azure Functions Startup Trigger

Eugen Podaru — Sun, 15 Nov 2020 14:30:30 +0000

Ever wanted to run some heavier initialization when your function app starts, but could not do it in the Startup class? Well, in this article I will walk you through creating a simple Azure Functions trigger that fires only once, exactly when you need it …🥁… when the runtime starts.

If you'd rather see the code, and skip the explanation, you can go straight to Github and check it out. Or if you prefer to just use it, and trust me that it works, you can go to Nuget and take it from there.

I went through the trouble of creating a proper library and put it on Nuget, because I am using it in one of my work projects, and I believe it is not such an uncommon scenario. Even if you do not have a use case for this, you might still want to go through the article since it is a great starting point for writing your own custom triggers and bindings for Azure Functions. It has all the pieces that you need to put together for such and endeavor with little extra code.

As for my use case, one of the solutions I am working on has a backend plane made of a mix of web APIs and function apps. In order to send messages between them, we use a thin wrapper on top of the Azure Service Bus. We have a single topic (one per tenant) in the Azure Service Bus namespace where all the publishers put their messages, and whoever is interested in a particular message type, can subscribe to it. Publishing messages is easy, since the topic is created, and its name is known in advance. Consuming the messages is a bit more complicated, since different services are interested in different types of messages, and each type of message has its own handler.

In the web APIs we solved this by having some helper code run on startup that goes through all the relevant assemblies and gathers the message type handlers that the service defines, and then creates the corresponding subscriptions on the topic.

In the function apps we could have had a function per message type. However, this was not acceptable for multiple reasons. First, the Azure Service Bus binding does not support managed identity yet. Second, the binding does not create the subscriptions if they do not exist, so that means that for every new message type, we would have had to create all the required subscriptions in advance. And last, why have some different approach, when the solution for web APIs works so great. The only problem was that we could not run the helper code in the Startup class, since the Startup class is meant for only setup and registration of services for DI.

I figured out that I could run the helper code in a function that is triggered at startup. Without knowing how the bindings and triggers work, I knew it was possible, since the Timer trigger has the RunOnStartup attribute which does exactly that.

How does it work?

Let's start with the trigger, which is the attribute the marks the function that should run at startup. You can see the code in the following listing:

Nothing to see here, move along!

Since the trigger is to be used on a parameter of a function, let's see the parameter type next. A trigger can support multiple parameter types, see the QueueTrigger for example, but it would be overkill for this particular case:

I added some common sense properties of the function app to the StartupInfo that can be useful in the triggered function. In my case, I use the name and the version, for example, to namespace the subscriptions, so that multiple services can subscribe to the same message types.

And with that we are done with the visible parts of the trigger. Next, let's see how to register the trigger with the runtime.

When the runtime starts, it scans all the linked assemblies for the WebJobsStartupAttribute assembly attribute which points to a class that implements the IWebJobsStartup interface. Once it gathers all these implementations, it calls Configure on each.

This is exactly how the Startup class is discovered as well, since FunctionsStartupAttribute actually inherits from WebJobsStartupAttribute and FunctionsStartup is an abstract class that implements IWebJobsStartup.

Here is the relevant code for how I register my extension:

The only thing that I do in the Configure implementation is call the AddExtension extension method of the IWebJobsBuilder. AddExtension expects an implementation of the IExtensionConfigProvider interface, which you can see in the following listing:

In the IExtensionConfigProvider I can already make use of DI, so in the above implementation I get some IOptions. These should be registered in the Startup class.

The really important thing that happens here is actually registering the StartupTriggerAttribute as a binding by calling the AddBindingRule and then connecting the binding with an implementation of the ITriggerBindingProvider by calling the BindToTrigger method. If you are writing a custom input or output binding you would be calling BindToInput or BindToOutput instead.

The ITriggerBindingProvider implementation is where I create the actual binding instance. The TryCreateAsync method of the provider is called for all the parameters of the function, but it returns a binding only for the ones that have the binding attribute applied and have a type supported by the binding. Since I only have to deal with parameters of type StartupInfo, the code is really simple:

The binding is an implementation of ITriggerBinding and it is where the heavy lifting is done. The CreateListenerAsync is called by the runtime to create a listener for the events that trigger the function. The BindingDataContract is how the binding exposes binding parameters that can be used in the binding expressions of other parameter attributes, and BindAsync is called by the runtime when it needs to bind to a parameter for a function invocation. The input parameter value might not be of the type expected by the trigger parameter, so this is where any conversion should happen. Finally, this method wraps up the parameter in an ITriggerData to be used by the runtime for the function invocation. Following is the binding implementation:

And finally we got to the listener which implements the IListener interface and is the place where listening for the trigger event source happens. In my case, the only event I am interested in, is the one happening when StartAsync is called, so I immediately invoke the function, no need to wait for any other event to happen. One thing to notice is that the listener uses the singleton attribute which ensures that only one instance of the listener is running regardless of the number of function app instances:

It is worth mentioning that the StartAsync method of the listener will be called when the runtime starts. But when does the runtime start? It turns out that it starts either when the function app starts, restarts, scales up or wakes up after being idle. So keep that in mind and make sure it fits with your use case.