DEV Community: Matt Allford

Beyond skeleton pipelines: who owns your software pipeline?

Matt Allford — Thu, 14 Aug 2025 00:41:08 +0000

Your current software delivery processes are probably working fine. They build your code, maybe run a test or two, and get your software to production. That's no small achievement, but here's a question that might make you pause.

When did someone last spend time improving them?

If you're like many software teams I've worked with, the answer is probably "not recently". Maybe not ever.

I've worked with developer teams at different maturity levels and have repeatedly seen this pattern. Great development teams who are busy building software but aren't large enough to have platform teams or dedicated operational folk responsible for the software delivery process. They typically start with a template from their cloud provider, delivery tooling vendor, or, these days, their favorite AI LLM. Initially, it gets the job done, but I refer to these as skeleton pipelines—the bare minimum framework to ship code.

Skeleton pipelines are a great starting point. They get you from nothing to shipping code quickly, but your pipelines are living, breathing artifacts. Before long, they need some meat added to the skeleton to provide functionality beyond the basics and make your delivery process robust and delightful.

I've seen when teams experience friction with their software release process, but don't treat it as a bug or feature improvement that needs to be logged, prioritized, and worked on. If your deployment pipelines are responsible for delivering your product to customers, shouldn't you treat them as part of your product? When was the last time someone added a pipeline improvement to your backlog or spent time making your deployment process better?

The important shift is treating delivery process improvements as regular development work, not spare-time projects that inevitably get pushed aside.

The ownership vacuum

Ask any group of engineers, "Who owns your software delivery pipelines?" and I bet you'll get different answers. When writing this post, I put that question into a search engine and reviewed the discussions, which weren't surprising. Some say developers should own what they build, others insist it's up to platform or DevOps engineers. Then there's the view that everyone is responsible because DevOps is about everyone working together. My observation is that often, no one owns it, and that's when it becomes a problem.

The distinction that matters isn't really about ownership. It's about accountability. Someone needs to be accountable for ensuring your delivery process evolves, improves, and supports your organization with shipping software rather than being a friction-filled process people loathe working with. Without that accountability, pipelines can easily become unmanaged components that gradually deteriorate until they become technical debt and a bottleneck instead of an enabler.

There's no one-size-fits-all answer for who should own and be accountable for your pipelines. You must discuss and agree based on your maturity, size, team structure, and individual skill sets. The important thing is that it is discussed and agreed upon, and not left to assumptions. Ask yourself now if everyone on the team knows how and where to raise an issue with the deployment process today. If the answer is no, then start there. Determine who's responsible and establish clear channels for pipeline feedback, problems, and improvements.

The cost of skeleton pipelines

This ownership vacuum creates predictable problems I've seen many times. Teams fall into the "if it ain't broke, don't fix it" mentality, leaving pipelines as-is for months or years. The issue is that what worked well 6, 12, or 18 months ago likely isn't fit for purpose today. New tooling and techniques exist now that didn't then. Your team will likely ship more frequently—or at least want to—and handle more complex deployments than when you first created that pipeline. The infrastructure you're deploying to has likely changed. Different developers and engineers might work at your company who didn't before. You're missing out on tooling and practices that could:

Save weekly hours
Bring new functionality to your software delivery lifecycle that adds business value
Catch issues before they become customer problems

Worse, they often become single points of knowledge. Whoever initially sets up the pipeline is the only person who understands how it works. When that person leaves or gets pulled onto other projects, your team inherits a mysterious system everyone fears touching.

Fragile pipelines break at the worst possible moments. They miss opportunities to catch issues early, optimize build times, have predictable outcomes, or provide better feedback to developers. Maybe, most importantly, they frustrate your team and slow down what they're supposed to enable: shipping great software and products.

You can't afford to ignore this

We're too busy building features to worry about pipeline improvements.

Sound familiar? It's a common pushback I hear from teams, and I get it. When trying to ship products and keep customers happy, spending time on what feels like an internal process is often considered a luxury you can't afford.

Platform teams are getting a lot of attention lately, and for good reason, as they work well for larger organizations. But if your company doesn't need everything else a platform team typically handles, creating one just for pipeline ownership is unnecessary. The reality is that you don't need a dedicated platform team to own your pipelines. You can borrow their mindset by treating your delivery process as part of your product. Pipeline improvements get backlog items, performance issues get bug tickets, and deployment pain points get the same attention as user-reported problems.

Start by paying attention to where your process hurts. If deployments make you nervous, your team loathes "release day", when releases fail for mysterious reasons or when you're manually testing things your team can automate. These pain points are your roadmap for high-value improvements.

Making the investment

Improvements may include:

Adding vulnerability scanning during builds
Configuring integration tests against ephemeral infrastructure
Properly storing and versioning your deployment artifacts
Implementing flexible approval workflows
Creating a robust deployment strategy
Measuring deployment metrics

I'm not giving you a shopping list of things to do. Instead, I recognize that "you don't know what you don't know" because I've been there too, and you may be experiencing issues you didn't know there were solutions to.

We have a white paper titled The ten pillars of pragmatic deployments. I bring this to your attention not for a checklist of everything you should do but for insight into what we think good deployments look like based on years of experience building a deployment automation tool. The ideas discussed in this post might help you enhance your software delivery process today or provide insight into issues and solutions you may not have been aware of.

Take ownership

It's easy to overlook software delivery pipelines. They're usually out of sight until they get in your way, kind of like plumbing. It's invisible when it works and catastrophic when it doesn't. Excellent plumbing supports everything in your house to function smoothly, and great delivery processes do the same for your product development.

Claim ownership of your software delivery processes and ensure someone is accountable for their care and feeding. Talk to your engineers involved in developing and releasing software, and find out where the pain is. Implement easy-to-use feedback loops, add pipeline improvements to your backlog, and treat deployment pain points as bugs worth investing in and fixing.

Your future self and team will thank you when your next deployment goes smoothly instead of keeping everyone up after hours or nervously huddling around a few pizzas doing your monthly release with crossed fingers.

What GitOps changes about elevated access

Matt Allford — Wed, 30 Jul 2025 23:41:26 +0000

Recently, we surveyed the industry to gain insights into the adoption and challenges of real-world GitOps, with the results forming the State of GitOps report. While reviewing the trends and results, the data around one key finding jumped out at me:

GitOps reduces elevated access

Overall, 66% of respondents agreed. Among organizations with higher GitOps maturity, agreement rose to 77%, but interestingly, there was also a slight increase in disagreement among the highest performers.

So, does GitOps change or reduce the need for elevated environment access, and should it?

How GitOps shifts access away from infrastructure

The 4 GitOps principles encourage you to avoid manually logging into environments and performing tasks. The goal is to manage system state declaratively through version-controlled configuration and apply those states automatically using reconciliation agents.

If fully embraced, GitOps agents continuously observe the system state, correcting any drift by reapplying the desired state defined in version control. Continuous reconciliation discourages manual changes and automatically reverses them if they occur.

In traditional operations teams, access to systems with SSH or tools like kubectl is common, but operating in a mature GitOps model should make these break-glass exceptions.

Most survey respondents agree that GitOps should reduce the need for privileged production access by replacing hands-on changes with declarative, auditable workflows.

Rethinking what elevated access means with GitOps

So why might some respondents to the survey disagree that embracing GitOps reduces elevated access?

At lower maturity levels, teams may treat Git as a safe, developer-focused system. But as Git becomes the source of truth for managing production infrastructure and applications, a key question emerges:

If a developer can change a manifest in Git for a production environment, does that emulate elevated access?

In a GitOps model, Git becomes the control plane, meaning write access to a production configuration in Git, especially with automatic reconciliation, becomes a form of production access.

High performers likely consider GitOps secure when configuring repositories well and implementing mature environment promotion workflows. However, Git isn’t the only sensitive surface in GitOps architecture. You must also secure the reconciliation controller and supporting workflow or promotion tooling with the same level of rigor. Together, these systems form the modern delivery pipeline and deserve the same access scrutiny once reserved for direct infrastructure.

High performers likely aren’t ignoring or rejecting GitOps’ security benefits, but instead acknowledge that you must treat Git itself as a sensitive operational boundary now more than ever. It’s now one of several critical control points in the delivery process that need strict governance.

Interestingly, these same high performers agreed that GitOps reduces overall elevated access. So, the takeaway isn’t contradiction; it’s maturity. High-performing teams see GitOps as secure only when Git, the reconciliation controller, and supporting environment progression tooling are all treated as critical access points, with the appropriate controls in place.

Handling exceptions to the GitOps model

In the early stages of adopting GitOps, it’s common for DevOps engineers, platform engineers, and even developers to rely on familiar tools like kubectl to inspect or tweak clusters and applications directly. Especially in learning environments and environments early in the deployment process, like dev, this is precisely what they’ll do. This approach lets them rely on familiar tools while gaining confidence in GitOps practices.

The ultimate goal is for nobody to have direct access to production environments, even if read-only.

In the State of GitOps report, we surfaced six core practices of GitOps:

Declarative desired state
Human-readable format
Responsive code review
Version control
Automatic pull
Continuous reconciliation

You won’t embrace all of these GitOps practices on day one, and that’s ok. As we learned from the DevOps movement, embracing a GitOps infrastructure and application delivery approach involves continuous improvement, learning, tweaking, and time.

With that said, even at high maturity levels with GitOps where you implement many or all of the practices, there may still be valid scenarios where folks need elevated access. For example, emergency responses (break-glass scenarios) or debugging complex or unreproducible issues.

In these cases, it’s crucial to have:

Break-glass procedures. Documented steps for requesting, granting, and revoking temporary access. These should include automated expiration, require multi-party approval, and be used only in exceptional circumstances.
Audit trails. You must log every access event, recording who accessed and changed what, when, and why.
Controlled suspension of reconciliation. Sometimes, you must temporarily disable the GitOps agent to prevent it from overwriting a manual change made during incident response. You should log and time-bound this suspension, and include procedures to reconcile safely afterward.
Post-incident recovery. After manual intervention, have a defined process for reconciling the system back to the desired state, committing required changes to Git, and re-enabling automatic reconciliation.

These situations should be rare. Ideally, releases have already passed through multiple environments that closely mirror production in both infrastructure and configuration. If you experience an issue where you find your developers or engineers need to access environments directly, after the issue gets resolved, question why. Could they have achieved the same outcome by using other systems and tooling? Is there an element of education and training required? Is there a need to implement supplementary tooling or processes to help troubleshoot and resolve issues that surface after deployment? Work on fixing it, and implementing that fix in your process so it doesn’t occur again.

Releasing to production shouldn’t be a production.

The evolving access conversation

GitOps won’t immediately eliminate the need for you to have a process for elevated production access, but it should help make it an exception rather than a routine operation. You must treat Git and GitOps tooling with the same care and control as production infrastructure. The fundamentals still apply; they apply in more places and different places than you’re used to.

As you mature your GitOps adoption, the conversation shifts from “do we still need access?” to “are we securing the new access surface and tooling?”.

Happy deployments!

Azure Functions: Managing Authentication and Secrets

Matt Allford — Wed, 06 May 2020 06:37:07 +0000

Prerequisites
Step 1 — Configuring a Managed Identity and Role Based Access Control
Step 2 — Creating a Cost Center Function
Step 3 — Accessing Secrets from Azure Key Vault
Conclusion

Introduction

Azure Functions provide flexibility with your workflows and logic processes, no doubt about it. In parts one and two of this blog series, you created different types of Azure Functions, provided input data with a trigger, configured event-based triggers and used output bindings to send data to other applications.

To date in this series, you have not worked directly with Azure Resources using the Az PowerShell module within the function itself. This is because access to Azure resources from the functions have not yet been configured. Additionally, you have not been shown how to handle passwords or secrets you may need to leverage within the function, you definitely don't want to be storing those in plain text in the run.ps1 file itself.

In this guide, you will configure a managed identity on the function app and provide access to Azure resources for that identity using Azure role-based access control. Next, you will deploy an event grid function that shows this access working by setting tags on Azure resources based on logic within the function. Finally, you will learn how to securely access secrets stored in Azure Key Vault from within a function.

When you're finished, you'll be able to securely provide Azure functions with role-based access to Azure resources and leverage password and secrets in your functions that are stored in Azure Key Vault.

Prerequisites

Before you begin this guide you'll need the following:

(optional) Familiarity with PowerShell would be beneficial
An Azure Subscription, you can create a free account if you don't have an existing subscription
An Azure function app based on PowerShell to integrate with Azure and create a new function in. If you have the function app from post one in this series, you can use that. This post follows on from post two in this series and will be using the same function app cloudskills20200406

Step 1 — Configuring a Managed Identity and Role Based Access Control

There will be times in your functions that you want to read, modify and maybe even delete resources within your Azure subscriptions. By default, the functions you create do not have permission to manage any of your Azure resources, whether that be in the same or separate subscriptions. By configuring a managed identity and setting role-based access control, the function app (and therefore all functions within the function app) will have the permission to manipulate resources that you provided the managed identity access to.

First, log in to the Azure Portal, search for Function App and click on your function app. Our function app is called cloudskills20200406, yours will be called something different.

With the function app opened, click on Platform features and then click on identity.

In this post you will configure a system assigned managed identity. Toggle the status to On and click Save. You'll be asked if you want to enable the managed identity, click yes to that as well. Note that the managed identity being registered will be named the same as your function app.

After a few moments the Object ID for the managed identity will be shown on this page. Copy that to the clipboard as you will use it to assign a role to the managed identity using PowerShell.

You now need to assign a role for the newly created managed identity. In this example the managed identity is being assigned the contributor role on the subscription. You may want to scope this further depending on your requirements, such as targetting a resource group rather than an entire subscription. To find your subscription ID, use Get-AzSubscription. Open up Cloud Shell and run the following command to provide the managed identity object the contributor role on an Azure subscription:

New-AzRoleAssignment -ObjectId 8aebb402-14ab-4ce7-b5c9-b4122384ffa7 -RoleDefinitionName Contributor -Scope /subscriptions/b0d214f3-4b5b-45f6-a841-db43c23acbba/

In this step you configured the function app to use the managed identity and created a new role assignment in Azure to provide the managed identity with access to resources. You now can manage objects in Azure from within your functions.

Step 2 — Creating a Cost Center Function

In an earlier blog post in this series you created an event grid function, which was triggered following a particular event, processed some data then wrote the processed data out to a table in Azure Storage.

In this step you will create another event grid function, but instead of writing out to another application or service, the function will leverage access from the newly assigned role in the previous step to modify an Azure resource directly. The function will look for when a new resource group is created, and based on the value of a tag named Env, will then add another tag to the resource group named CostCenter with an appropriate value, either 0001 if the Env tag is Development, or 0002 if the Env tag is Production.

This paragraph is going to be high-level steps as you went through this in the previous post in this series. If you haven't read that post, please refer to it for the detailed walk-through steps. In the Azure Portal, open your function app and create a new function based on an Azure Event Grid Trigger. Add an event grid subscription with the following settings:

Topic Types: Azure Subscriptions
Filter to Event Types: Resource Write Success

Next, after your event grid subscription is created, you will be back in the editor looking at the run.ps1 file. Copy the PowerShell function below and click Save.

param($eventGridEvent, $TriggerMetadata)

$costCenterDevelopment = "0001"
$costCenterProduction = "0002"

$eventSubject = $eventGridEvent['subject']

if ($eventSubject.contains('resourcegroups')) {
  $rg = Get-AzResourceGroup -Id $eventSubject -ErrorAction SilentlyContinue

  if($rg -and $rg.Tags['Env']){
    switch($rg.Tags['Env']){
      "Development" {
        $rg.Tags.Add('CostCenter', $costCenterDevelopment)
        Set-AzResourceGroup -Name $rg.ResourceGroupName -Tag $rg.Tags
      }
      "Production" {
        $rg.Tags.Add('CostCenter', $costCenterProduction)
        Set-AzResourceGroup -Name $rg.ResourceGroupName -Tag $rg.Tags
      }
    }
  }
}

Notice in this code the PowerShell Az cmdlets are being leveraged to Get and Set Resource Groups. By default, when a PowerShell function is created, behind the scenes there are some extra files. You will explore those further in a later blog in this series, but one of those files is requirements.psd1, which contains the following code:

# This file enables modules to be automatically managed by the Functions service.
# See https://aka.ms/functionsmanageddependency for additional information.
#
@{
    'Az' = '3.*'
}

What this means is that the Az PowerShell modules are loaded with the function and are available for use.

Finally, go ahead and create a test empty resource group so the new function is triggered. Make sure during creation you assign a tag with a Name of Env and the value is either Development or Production. You can use the PowerShell example below to create a resource group with an appropriate tag:

New-AzResourceGroup -Name costcenter-rg01 -Location 'West US' -Tag @{Env="Development"}

After a few moments, if you watch the log stream you will notice the function is triggered. If you check the tags on your resource group, you now will have an additional tag with a name of CostCenter.

This capability provides you with effectively an unlimited range of options on configuring role-based access control to your Azure resources to then manage and maintain them using Azure Functions. In the final step of this post, you will learn how to access secrets stored in Azure Key Vault such as passwords and API keys securely from your function.

Step 3 — Accessing Secrets from Azure Key Vault

Occasionally you will want your function to connect to an endpoint that it is not authorized to by default. This could be an Azure resource, such as the guest operating system of a virtual machine, or it could be an external service such as an alerting system. Hopefully it goes without saying, but just in case, it is not a good idea to save these secrets in the run.ps1 file in clear text. Historically, the method was to create a new "application setting" within the function app which has a name and a value. The application settings can be called from the PowerShell function by using an environment variable such as $env:AppSettingName. Application settings are encrypted at rest and are transmitted over an encrypted channel, but they can still be exposed in plain text and there are no advanced secrets management capabilities such as versioning or auditing.

In late 2018 Microsoft announced support for accessing Azure Key Vault secrets from a function app. The good news (for backward compatibility) is that it still uses the function application setting, instead of putting the secret itself in the value, you enter a reference to the Azure Key Vault secret.

In this step you will add an application setting and store the value directly in the application setting. Next, you will create a new Azure Key Vault, provide the managed identity we created in Step 2 access to the Key Vault, create a new secret within the Key Vault and finally, update the value of the application setting to reference the Key Vault Secret instead of storing the value directly.

First, select your function app in the Azure Portal, click on Platform features and then click on Configuration.

Click New Application Setting, for the Name enter AlertSystemAPI and for the value enter KeUkcwjdBIrmgEzvKOlRuNKowAfg and then click ok. This isn't a real key for anything, it is simply an example.

On the configuration screen, click on save and click continue. At this point in time, you would be able to reference the value of this new application setting by using $Env:AlertSystemAPI in your PowerShell code inside of run.ps1.

Next, you will create an Azure Key Vault. The examples below are done in PowerShell to make it easier than navigating through the portal with screenshots. You might try it as well using PowerShell from the Cloud Shell, or https://shell.azure.com.

New-AzKeyVault -Name CSFunctionVault -ResourceGroupName functiondemo -location "Australia East"

By default the Key Vault doesn't have an access policy assigned, so no users or accounts can access the vault. Here, you will provide the managed identity of the function app (the ObjectID being used below is the same one we used in step 1) access to get secrets from the new vault, as well as providing your account access to get and set secrets. Just make sure to update the value of ObjectId and -UserPrincipalName in the text below:

Set-AzKeyVaultAccessPolicy -VaultName CSFunctionVault -ObjectId 4f14fcc6-b345-4f8c-9589-5d21edd0772c -PermissionsToSecrets get -PassThru

Set-AzKeyVaultAccessPolicy -VaultName CSFunctionVault -UserPrincipalName your.name@domain.com -PermissionsToSecrets set,get -PassThru

Finally, you will create a new secret in the vault with a name of AlertSystemAPI and a value of KeUkcwjdBIrmgEzvKOlRuNKowAfg:

$secret = ConvertTo-SecureString -String 'KeUkcwjdBIrmgEzvKOlRuNKowAfg' -AsPlainText -Force

Set-AzKeyVaultSecret -VaultName CSFunctionVault -Name AlertSystemAPI -SecretValue $secret

In the output of the last command, you will see an ID for the secret that has been created. Copy this ID to your clipboard, you will need it in a moment.

Lastly, you will now update the value of the application setting in the function app to reference the secret you just created. In the Azure Portal, navigate back to your function app, click on Platform features and Configuration. Click on the Edit button for the entry named AlertSystemAPI and replace the value with the following, making sure to replace <SecretID> with the ID that you copied in the last step:

@Microsoft.KeyVault(SecretUri=<SecretID>)

For example:

You still reference this value by using $Env:AlertSystemAPI in your PowerShell code inside of run.ps1, but now instead of the value being stored in the function app, it is stored in a central Key Vault providing the standard and regulations of secrets management.

Conclusion

In this article you configured a managed identity on a function app and configured an Azure role assignment to a subscription for the managed identity. This provides the ability for functions within the function app to manage Azure Resource that is has delegated access to. Next, you deployed another event grid function and used logic within the PowerShell function to put a cost center tag on a new resource group, dependant on which environment tag value was assigned to the resource group during deployment.

Finally you reviewed the application settings within a function app to understand how environmental values are stored and called. Though these are encrypted at rest and in flight, they lack some core secrets management capabilities, so you enhanced this functionality by creating an Azure Key Vault Secret in a new Key Vault and referenced the secret from the function app application settings.

Next up in this series you will look at developing and running functions locally in Visual Studio Code, followed by deploying the function to Azure directly from Visual Studio Code. Finally you will create a new Azure function app that integrates with a git repo in Azure DevOps and uses a CI/CD pipeline to deploy updates to the function app.

Azure Functions: Creating a PowerShell Event Based Function

Matt Allford — Sun, 03 May 2020 10:42:15 +0000

Introduction

In the first post in this series, you created a basic Azure Function based on a HTTP webhook trigger and learned some of the fundamental concepts of Function Apps and functions. While webhook functions certainly are valuable and solve problems, some powerful workflows can be created that are based on event-driven data.

In this guide you will create a new function that will integrate with Azure Event Grid, resulting in the function being triggered when a particular event takes place. You will analyze the data being sent to the function by event grid to understand what you are working with.

You'll also set up an Azure Storage account to use as an output binding from the function. When the function is triggered, you will use logic to get data from the triggered event and insert that into a new row within Azure table storage in the destination storage account.

The goal in this guide is to trigger a function when a file is uploaded to blob storage in a specific storage account. If the file is a CSV file, the function will collect some data from the incoming event data, specifically, the file name of the CSV that was uploaded and using an output binding, will insert a new row into an Azure Storage Table with some data from the event.

When you're finished, you'll have a good understanding of how event-driven data can be used with Azure Functions and have exposure to creating output bindings to send processed data in the logic of the function to external components.

Prerequisites

Before you begin this guide you'll need the following:

(optional) Familiarity with PowerShell would be beneficial
An Azure Subscription, you can create a free account if you don't have an existing subscription
An Azure function app based on PowerShell to create a new function in. If you have the function app from post one in this series, you can use that.
A Storage account where blobs will be uploaded to be the event grid trigger. In this demo I created a separate resource group and storage account for this by running the following PowerShell commands from Azure Cloud Shell:

New-AzResourceGroup -Name BlobStorage -Location "Southeast Asia"
New-AzStorageAccount -Name csfunctionstg01 -ResourceGroupName BlobStorage -Location "Southeast Asia" -SkuName Standard_LRS

Step 1 — Creating an Event Grid Based Function

This guide uses the function app created in the previous post in this series. If you don't have a function app, refer back to that post to create one for use here.

First, log in to the Azure Portal and search for Function App. Our function app is called cloudskills20200406, so click on that. Yours will be called something different.

In the panel on the left-hand side, click the + next to Functions to create a new function.

Scroll down and select the Azure Event Grid trigger and give the function a name. Ours will be called CSEventGridTrigger. Click Create to create the new function.

You will be taken to the PowerShell code editor for the new trigger. You will notice there is a parameter named eventGridEvent. This is the variable that will contain the incoming event data, which we'll need to explore after configuring an Event Grid subscription.

At the top of the window, click on Add Event Grid subscription.

On this page you are presented with some forms to fill out to create the event subscription. This will be looking at the storage account we created in the prerequisites section of this guide. A few things to note:

You can optionally create filters and additional features of the event subscription to further target which events you are interested in, to trigger the function. For example, you could create a filter here so that only events where a CSV file was uploaded to blob storage are delivered to the function. You'll be doing this logic in the PowerShell function itself, this is just an example
For the topic type, observe the different resources that can be selected within an event subscription
On the event type, in our example this is filtered to just show Blob created as this is the only event we are interested in having trigger the function

Fill out the basic information for the event subscription and then click on Create.

You've created a new PowerShell function and configured an Event Grid subscription to send data to this new function based on particular event types.

Step 2 — Reviewing Event Grid Data Sent to the Function

Now that you have an event grid subscription created, you need to understand what data and properties are being sent to the function by event grid. Without this information, it will be difficult to maniuplate the data and write logic into your function.

In the template that was created for us, the $eventGridEvent variable is being output to the host, so if we trigger the function we can see the data in the Logs console. An alternative method is to convert the object to JSON and then output that, giving us the full data structure to copy into a text editor to view the data structure and properties.

Replace line four with the following code and click save. Your function should look the same as the image below.

$eventGridEvent | ConvertTo-Json | Write-Output

Click on Logs at the bottom to reveal the log-streaming service. Click on Expand to make the window larger. You will come back to this window in a moment to review the output.

Next, you will perform an event that causes the function to trigger and review the event data. Open another browser tab or window with the Azure Portal open and navigate to the storage account that was created in the prerequisites section of this guide. After clicking on the storage account, use the search feature in the properties pane and search for Containers.

Click + Container to add a new container. In this guide, the container is called test and has the default private access.

Click on the newly created container and then click upload to upload a small test file from your computer, such as a text file. Click Upload to upload the file.

Switch back to the window or tab where we left the log streaming running within the function. After a few moments you will see the function has executed. In the log-stream output you will see a JSON body showing the properties and data sent to the function by the Event Grid service, as shown in the image below (the highlighted text is the JSON data). You can copy this data to a text editor to observe as you build out the logic of your function. For example, there is a property in this JSON output named subject. To reference or use that in the function itself, you can use now use $eventGridEvent.subject.

When you are finished, close the log streaming console (or collapse it) so we can see the run.ps1 script again. Leave your other tab or window open with the storage account where you uploaded the blob, that will be used again later in this guide.

Step 3 — Configuring an Azure Table Storage Output

When you come up with an idea for a function to solve a problem or add value, the function itself often won't be the end of the line. You will find requirements for your function needing to send data to another application or system, whether that be for logging purposes, triggering another downstream process, creating an alert, and so on. Azure functions have a concept called output bindings, which are preconfigured connectors provided to you so you don't need to code the connection in manually from your function. In this step, you will create and leverage an output binding to write data from the function in to a row in Azure Table Storage.

First, you need to create the output binding for the function. In this example, you are going to output the result to a new Table named FunctionOutput hosted in the same storage account that is already being used by the function app. You probably wouldn't normally do this, it is just to save creating another storage account as we already have one we can use.

In the left-hand side panel under your function, click on integrate. Under Outputs, click on + New Output, select Azure Table Storage and click Select.

Change the Table Name to FunctionOutput. Notice the parameter name is set to outputTable, this is what will be used later in the PowerShell function output binding. Leave the default Storage account connection as this will automatically select the storage account associated with the function app, but if you did want to select a different storage account as the target, you can click new and select an alternative storage account (or create a new one). Click on save to save the output binding.

In the left-hand side panel, click on your function name to be returned to the run.ps1 script where we will bring everything together with some PowerShell logic to pull some properties from an incoming event grid trigger and output them to Azure Table Storage.

Step 4 - Putting Logic in the PowerShell Function

Let's do a quick recap of where we are at. So far we have:

Created a new PowerShell function based on an Event Grid trigger
Created an Event Grid subscription which is looking for a Blob Created event and will send the event data to the newly created function
Triggered the event by uploading a test file to the blob storage and viewed the contents of the event data that is sent to the function
Created a binding output based on Azure Table Storage to a separate storage account

Finally, you will put some logic into the run.ps1 file to process the event grid data and then use Push-OutputBinding to send the processed data to Azure Table Storage.

Below is a small example of what you will run in the PowerShell function. Copy and paste this into the run.ps1 file in the Azure Portal and click Save.

param($eventGridEvent, $TriggerMetadata)

Write-Host "New event occurred - $($eventGridEvent.eventType)"

$SubjectName = $eventGridEvent.subject -split "/"
$FileName = $SubjectName[-1]

if ($Filename -like "*.csv") {

    $Data = [PSCustomObject] @{
    partitionkey = $eventGridEvent.eventTime
    rowkey = $eventGridEvent.id
    FileName = $Filename
    Action = $eventGridEvent.data.api
    }

    # Associate values to output bindings by calling 'Push-OutputBinding'.
    Push-OutputBinding -Name outputTable -Value $Data
}

Let's break down some components of the PowerShell code being used in this example.

Firstly, you define some parameters to be used within the function. $eventGridEvent is the name of the parameter that will store data from the inbound Event Grid subscription. Then Write-Host is being used just to show an example of writing some information to the log stream.

param($eventGridEvent, $TriggerMetadata)

Write-Host "New event occurred - $($eventGridEvent.eventType)"

Next, it is going to get the data stored in $eventGridEvent.subject and split it based on the forward slash. You can tell that this needs to be done by examining the output data from step 2. After the data has been split, we are looking for the file name which will always be the last item, which can be accessed by using [-1]. The filename is stored in a variable named $FileName.

$SubjectName = $eventGridEvent.subject -split "/"
$FileName = $SubjectName[-1]

Finally, if the file name ended in .csv, a custom PowerShell object will be created. Data is being pulled from the Event Grid data by accessing properties of $eventGridEvent as well as defining the file name that we extracted in the previous step. The partitionkey and rowkey values are specific to Azure Table Storage and these provide unique IDs to that row of data. Conceptually you do not need to worry about these, but if you are following along and weren't familiar, you may have wondered why those properties were included.

The last step is to use Push-OutputBinding to send the data in the custom object to Azure Table Storage, specifically the endpoint we defined in Step 3.

if ($Filename -like "*.csv") {

    $Data = [PSCustomObject] @{
    partitionkey = $eventGridEvent.eventTime
    rowkey = $eventGridEvent.id
    FileName = $Filename
    Action = $eventGridEvent.data.api
    }

    # Associate values to output bindings by calling 'Push-OutputBinding'.
    Push-OutputBinding -Name outputTable -Value $Data
}

Step 5 - Triggering the Function and Reviewing Output

Using Azure Portal, go to the storage account you created in the prerequisites section of this guide. You should still have this open in another browser window/tab from Step 2. It will have a container and a test file already from step 2. Upload another document by clicking Upload, select a CSV file from your computer and then click Upload again.

Optionally, navigate back to the log stream console of the function to monitor the function being triggered.

If you have Azure Storage Explorer installed, you can use that to connect to the storage account that is being used by the function app and look at the table storage. There should be a new table named functionOutput with a row of data that was inserted by the output binding.

Not everyone may have Azure Storage Explorer installed and configured and it is OK if you don't, though it is a great utility to have. In this next step you will use PowerShell in the Azure Cloud Shell to review the output data.

Open a new Cloud Shell, either by clicking on the cloud shell icon in Azure Portal, or in a new browser window or tab navigate to https://shell.azure.com. If you still have the tab open from a previous step, you can use that but you may need to reconnect your session. You can use the code below to look at the data in table storage of the same Azure Storage Account that is being used by your function app. Just be sure to change the name of the resource group hosting your function app specified in $ResourceGroup below.

The code below assumes you used the default storage account within the function app for the output binding and that there are no other tables in that storage account.

# Install the aztable Ps Module
Install-Module aztable

# Get the storage key for the storage account
$ResourceGroup = "functiondemo"
$storageaccount = Get-AzStorageAccount -ResourceGroupName $ResourceGroup

$storageAccountKey = (Get-AzStorageAccountKey -ResourceGroupName $ResourceGroup -Name $storageaccount.StorageAccountName).Value[0]

# Get a storage context
$stgcontext = New-AzStorageContext -StorageAccountName $storageaccount.StorageAccountName -StorageAccountKey $storageAccountKey


$table = (Get-AzStorageTable -Context $stgcontext).CloudTable

Get-AzTableRow -Table $table | ft

You can see the data that we collected in our custom object inside the function has been inserted in to the Azure Storage table as a new row.

Conclusion

In this article you created a new PowerShell function based on the Azure Event Grid template. You then configured an Event Grid Subscription to listen for an event type of Blob created on a particular storage account, which would send the event data to the PowerShell function. Next, you tested the trigger and observed the contents of the event by outputting it to a JSON object, viewable in the function log stream.

Afterward, you configured an output binding for Azure Table Storage using the default storage account that is created with the function app. You then updated the code within the run.ps1 file which inserted a subset of selected data to Azure Table storage if a CSV file is uploaded to the source blob storage account. You then uploaded a CSV file to trigger the logic from end-to-end and finally reviewed the output in Azure Table Storage.

While the specific example used in this guide may not provide any immediate real-world value, the purpose was for you to observe and understand the power of event-based functions and explore methods to have the function pass this data on to another system or process via an output binding. The options are effectively limitless and now you can put your imagination to the test to build functions, adding value to you and/or your customers.

In the next post in this series, you will create another event-based function and explore the ability to have the function authenticate and modify Azure resources and access secrets from Azure Key Vault.

Azure Functions: An Introduction for DevOps Engineers

Matt Allford — Tue, 21 Apr 2020 00:48:17 +0000

Prerequisites
Understanding Function App Plans
Step 1 — Creating a Function App
Step 2 — Creating Your First Azure Function
Step 3 — Reviewing the Function in the Azure Portal
Step 4 — Running the Function
Conclusion

Introduction

Azure Functions is a serverless compute service provided by Microsoft, enabling you to run small pieces of code on-demand without a need to build the underlying infrastructure to run that code. You can write code in a variety of languages and publish these blocks of code as functions, which is then hosted in Azure in a container called a Function App. Functions run when they are "triggered" and you'll see that in action in this guide. Though functions can run for a longer time under the premium and app service plans, the ultimate goal of a function should be to do a particular task and do that task as efficiently as possible.

In this first part of a multi-part guide, you will review Azure Function App plans and deploy an Azure Function App. Next you will examine the high-level components of the Function App, deploy a PowerShell based function and review the logic of a templated PowerShell webhook function. Afterwards, you will trigger the function using several different methods and observe the output and log stream. When you're finished you will have a PowerShell based Azure Function that you can modify to accept different types of input and perform actions when the function is triggered and have a fundamental understanding of what capability Azure Functions can provide.

By the end of the series, you will have deployed functions based on different triggers, worked with output bindings to send processed data to other applications, understood how authentication and secrets management works, developed, tested and deployed functions in Visual Studio Code and finally, put the function code under management of git and built a CI/CD pipeline in Azure DevOps to deploy your function when new code is committed to a master branch.

Prerequisites

Before you begin this guide you'll need the following:

(optional) Familiarity with PowerShell would be beneficial
An Azure Subscription, you can create a free account if you don't have an existing subscription

Understanding Function App Plans

Like any cloud service, it is important to understand what the cost is going to be and what options are available for optimizing that cost. Three plan types can be selected when you create a function app and each option influences the scalability, resource availability and support for advanced features. The plan type cannot subsequently be changed, so ensure you understand the options below upfront:

Consumption. This is the default hosting plan when creating a function app. Billing is based on the number of function executions, execution time and memory used. You only pay for the time your functions are running and scale is handled automatically by Microsoft. At the time of writing, there is a free grant (per month) from Microsoft for the consumption plan, allowing up to 1 million executions and 400,000GB-s (seconds) of resource consumption. By default, functions running under a consumption plan timeout in 5 minutes but can be expanded to a maximum of 10 minutes.
Premium. You can specify several warm instances that are always running and ready to run your function. At least one warm instance is required at all times per plan, meaning there will be a minimum monthly cost even if you don't trigger any functions. With that said, it does mean there is no cold start like you will see when triggering functions on a consumption plan. Functions running under a premium plan have a default timeout of 30 minutes and can be set to run for an unlimited time. Premium plans also provide some advanced capability when compared to consumption, such as being able to connect the Function App to an Azure Virtual Network (VNet) for more controlled connectivity.
App Service Plan. An App Service Plan is a set of dedicated compute resources in Azure where other App Service apps, such as a Web App, can run. Microsoft provided the ability to connect Azure Functions to an App Service Plan, meaning you can use the underlying resources in the App Service Plan for multiple services. Typically you would use this plan if you already have an App Service that isn't fully utilized or if you want to provide a custom image that your functions will use when they run.

The biggest downside that is usually associated with the consumption plan is that the functions are cold, meaning there will be a small lag time between when the function is called and when the execution of the function is completed. You will observe this later in this guide. Think about what your function is doing and whether the response from the function needs to be as quick as possible. If another process or a user is waiting on the result of the function running, it might be worthwhile to look at using one of the other plans.

Step 1 — Creating a Function App

In this step you will create an Azure Function App using the Azure Portal selecting PowerShell as the runtime. This will provide the resources you need as the backbone to then create your first function.

First, it is important to understand the resources that are required when provisioning a function:

Function App. Think of this as the logical container that lets you group functions together. This will need a unique name in the Azure Cloud
Storage Account. Internally, functions use storage for operations of the function app

Optionally, you can enable Application Insights on the Function App to provide monitoring of the function app and functions.

Let's get stuck in and create your first function app. Log in to the Azure Portal and search for Function App and then click Create Function App.

Fill out the basic information for the function app and then click on Next: Hosting:

Subscription: Select the Azure subscription that you will deploy this function to.
Resource Group: We're creating a new resource group for this demo called functiondemo
Function app Name: This is the name of your function app and will be the URL used to access functions within your function app. As noted above, this needs to be globally unique.
Publish: There is capability to publish a function as a docker container rather than raw code, but for this demo select Code.
Runtime Stack: Use this to select the language that your functions will be written in. For this demo, use PowerShell.
Version: Depending on the language you select, there may be different versions available to use in the runtime. At the time of writing, version 6 is the only one available for PowerShell
Region: Select the Azure region where your function app will be hosted

On the next page we get some options for the hosting of the app function. Select your desired options and then click Next: Monitoring

Storage Account: A storage account is required for a function app. Either choose an existing account or create a new one. We're creating a new account for this demo.
Operating System: Depending on your chosen runtime you can select the underlying Operating System that your functions will run on. We're using Windows as it is the only option for PowerShell
Plan Type: As we covered in the introduction, this is where you select the plan for the function app. This can't be changed later. For this demo we're using the consumption plan.

You can optionally enable Application Insights and add tags to the resources being provisioned. We aren't looking at App Insights in this demo, so we've chosen not to enable it. Disable Application insights then click on Review + Create. Go ahead and review your selections and click Create to provision the resources.

Finally, let's look at what was provisioned. If you followed along you will have:

An App Service Plan
An Azure Storage account
An App Service

In the next step you will look at the App Service and create a function.

Step 2 — Creating Your First Azure Function

In this step you will create a PowerShell Azure Function using the Azure Portal.

Using the Azure Portal, search for Function App and click on the Function App you provisioned in the previous step. The Function App is currently empty, so click on either the + next to Functions on the left-hand side or click on the + New Function button to create a new function.

Some different tools and methods can be used to create and configure Azure Functions. One of the options is to create the function in the Azure Portal, which is suitable for getting started. In a later blog in this series we will look at using Visual Studio Code for local development and publishing, but for now select In-Portal and click Continue.

Microsoft provides several templates for functions to help you get started quickly depending on what you want your function to do. For this demo select Webhook + API and click Create.

A new function is created within the function app and you are presented with PowerShell code as a template for the function.

In the next step, we will review some important files that have been created as part of the function and break down the example PowerShell function to understand the workflow.

Step 3 — Reviewing the Function in the Azure Portal

At the bottom of the screen, click on the arrow next to Logs. The Logs window provides a streaming view of the logs which can be useful when wanting to debug or troubleshoot a function. you will see this in action when we run the function in the next step.

On the right-hand side of the screen, click on the arrow to expand a pane where you can view the files in the function.

Each function contains two important files that you need to be familiar with that you can see in the panel on the right hand side. You can click on each file to view it in the editor:

Run.ps1. This is the PowerShell function that is currently open in the portal editor. This is the code that gets run when your function is triggered.
function.json. This is a configuration file that defines the behavior of the function, such as how it gets triggered, what authorization is required and what input and output bindings are defined. This example defines a HTTP in binding, named Request, a HTTP out binding named Response and you will come back to these in a few moments when we break down the PowerShell function.

Speaking of authorization, three values can be used:

anonymous: No API Key is required and the function can be triggered by anyone
function: A function-specific API Key is required in the call to the function
admin: The master key is required in the call to the function

By default, functions use the function authorisation option.

Click on Run.ps1 if you don't already have it selected and let's break down the logic in the example function.

The first line is to ensure the System.Net library is being used in this function. Like a standard PowerShell function, some parameters have been defined:

Request: This is the name of the "HTTP in" binding defined in function.json and will contain the data coming in with the HTTP trigger, via a query or a body, and the data will be available in this variable in the rest of the script.
TriggerMetadata: This parameter is optional and can be used to access extra information about the trigger. The metadata available does vary from binding to binding, but they all contain a sys property with some data.

using namespace System.Net

# Input bindings are passed in via param block.
param($Request, $TriggerMetadata)

The following is an example of using Write-Host to output information to the Log stream. You will see this in the Log console when we run the function a little later on.

# Write to the Azure Functions log stream.
Write-Host "PowerShell HTTP trigger function processed a request."

In the following lines, we are looking to see if a Name value has been provided by the person or process that triggered the function, either as a query parameter or in a body send with the request.

If a value was provided for Name, it is stored in the variable $Name and the $status variable is set to [HttpStatusCode]::OK. The function would then keep running to the next block below. If a value for Name cannot be found on the incoming request, the $status variable is set to [HttpStatusCode]::BadRequest and a custom message is put in to the $body variable.

One way or another, when this piece of code completes, we will have values for $status and $body.

# Interact with query parameters or the body of the request.
$name = $Request.Query.Name
if (-not $name) {
    $name = $Request.Body.Name
}
if ($name) {
    $status = [HttpStatusCode]::OK
    $body = "Hello $name"
}
else {
    $status = [HttpStatusCode]::BadRequest
    $body = "Please pass a name on the query string or in the request body."
}

The final part of the function is to call Push-OutputBinding and provide the value to an output so it can be processed. For this example, the output being used is a HTTP output. For the Name parameter in this example, the value Response is being used. Remember a few moments ago where you looked at function.json and a HTTP binding with an out direction was defined named response? That is what is being called now and the data in the -Value parameter will be sent and processed by the output binding. You can create different types out output bindings and that will be explored further in the next post in this series.

# Associate values to output bindings by calling 'Push-OutputBinding'.
Push-OutputBinding -Name Response -Value ([HttpResponseContext]@{
    StatusCode = $status
    Body = $body
})

You can edit the function.json and run.ps1 files directly in the Azure Portal and click on Save at the top to save the file.

In the next step, you will get your hands dirty and trigger the function in a few different ways and observe the output.

Step 4 - Running the Function

Now that you understand the logic of the function when it is triggered, in this step you will trigger the function in a few different ways to observe the behavior. We can use the Azure Portal to trigger the function and observe the logs and the output.

In the Azure Portal, ensure you are still on the screen with the run.ps1 function and you have the logs visible from the panel at the bottom, and the Test area showing in the pane on the right. Your screen should look like this:

In the test area, ensure the HTTP method is set to GET and then under query, ensure there is nothing in the name or value fields (if they are showing) and then click Run down the bottom. Notice that the function takes a few seconds to complete. This is because of the cold start on the consumption plan. Because we didn't pass the function a name with a value, either in the query or in a request body, the function will fail and we will receive the response code 400 with a message telling us to pass a name.

Behind the scenes, this is the actual URL that was used to trigger function:

https://cloudskills20200406.azurewebsites.net/api/HttpTrigger1?code=jWiMZha0KyfCSUOEDUXb7F9TbOgMBCHIQP52nJqnoZdsZaKB2CBhMQ==

If we break that URL down further, this section is the FQDN of the function app and the name of the function we are triggering:

https://cloudskills20200406.azurewebsites.net/api/HttpTrigger1

And below, after ?code= is the function key that we need to append to the trigger, because the function is expecting this for authorization:

?code=jWiMZha0KyfCSUOEDUXb7F9TbOgMBCHIQP52nJqnoZdsZaKB2CBhMQ==

If the authorization in the function.json file was set to anonymous, we would not need to append this key. There are a few ways to get the key to use, the easiest way on the screen you are on is to click on the </> Get Function URL at the top of the screen:

Go over to the right-hand pane and this time click on Add parameter under Query. Enter name for the name, put your name in the value and click on Run again. This time you will see a Status 200 returned, and the output will display "Hello <yourname>\".

Behind the scenes, this is the URL that was used to run the function this time. You can see the query for name and the value appended to the end.

https://cloudskills20200406.azurewebsites.net/api/HttpTrigger1?code=jWiMZha0KyfCSUOEDUXb7F9TbOgMBCHIQP52nJqnoZdsZaKB2CBhMQ==&name=Matt

Next, instead of sending the name as a query, send it as a key value pair in a JSON body. Change the HTTP method from GET to POST and to clean up, click x to remove the query we used in the last step. In the Request Body, by default it will have a JSON payload with a key value pair of name and Azure. Leave that for this demo, and click on Run. Again, you will see a status 200 returned and the output will change to "Hello Azure".

Hopefully you've been looking at the Logs window every time the function has been triggered, though sometimes it can be a bit buggy. If not, we will examine it now. The first and third lines are informing us that the trigger has been called and has been executed. The second line is the output from line 7 in the Run.ps1 file, showing how we can use Write-Host within a function to output information to the log stream.

2020-04-06T09:56:12.450 [Information] Executing 'Functions.HttpTrigger1' (Reason='This function was programmatically called via the host APIs.', Id=29592acd-dd21-4cc0-b5de-ca9481a9148b)
2020-04-06T09:56:12.494 [Information] INFORMATION: PowerShell HTTP trigger function processed a request.
2020-04-06T09:56:12.494 [Information] Executed 'Functions.HttpTrigger1' (Succeeded, Id=29592acd-dd21-4cc0-b5de-ca9481a9148b)

Conclusion

In this article you have explored the capabilities of Azure Functions and reviewed the options available for running the functions in different App Plans.

You deployed a new Function App in Azure and created a new function based on a webhook trigger. Once the function app was available, you reviewed the authorization options and reviewed the logic provided in the default templated PowerShell function.

Finally, you triggered the function using several methods that provided varying results and observed the output both in the Azure Portal and in the Log Stream.

In the next post in this series, you work with PowerShell based functions in more detail, creating examples where the function will perform actions on Azure resources based on particular logic and output the results using a binding other than HTTP.

DEV Community: Matt Allford

Beyond skeleton pipelines: who owns your software pipeline?

The ownership vacuum

The cost of skeleton pipelines

You can't afford to ignore this

Making the investment

Take ownership

What GitOps changes about elevated access

How GitOps shifts access away from infrastructure

Rethinking what elevated access means with GitOps

Handling exceptions to the GitOps model

The evolving access conversation

Azure Functions: Managing Authentication and Secrets

Table Of Contents

Introduction

Prerequisites

Step 1 — Configuring a Managed Identity and Role Based Access Control

Step 2 — Creating a Cost Center Function

Step 3 — Accessing Secrets from Azure Key Vault

Conclusion

Azure Functions: Creating a PowerShell Event Based Function

Introduction

Prerequisites

Step 1 — Creating an Event Grid Based Function

Step 2 — Reviewing Event Grid Data Sent to the Function

Step 3 — Configuring an Azure Table Storage Output

Step 4 - Putting Logic in the PowerShell Function

Step 5 - Triggering the Function and Reviewing Output

Conclusion

Azure Functions: An Introduction for DevOps Engineers

Table Of Contents

Introduction

Prerequisites

Understanding Function App Plans

Step 1 — Creating a Function App

Step 2 — Creating Your First Azure Function

Step 3 — Reviewing the Function in the Azure Portal

Step 4 - Running the Function

Conclusion