DEV Community: Sogeti

Is low code replacing traditional development?

Peter Rombouts — Mon, 20 Jul 2020 06:12:59 +0000

Originally posted on SogetiLabs: https://labs.sogeti.com/is-low-code-replacing-traditional-development/

Spoiler alert; no.

Low code vs Traditional Dev

My colleague, friend and SogetiLabs Fellow Daniel Laskewitz and I frequently talk about this topic. His field of expertise as Microsoft MVP covers the Microsoft PowerPlatform including low code systems like Power Automate (formerly Flow).

All too often people see a division between low code and traditional development using languages like C#, Java, TypeScript and Go.

In the real world however, these systems work together perfectly.

Most of the times, you cannot solve a problem with only low code. Think about scenario’s where you should link to old legacy systems or complex API calls. In those cases low code without any enhancement cannot natively connect to those systems.

Behold custom connectors

In the Microsoft ecosystems, custom connectors allow you to bridge this gap. This way, the low code system can interact with any system you write a connector for. This may be common knowledge, but the fact is that most developers do not see how big this really is.

This means you can link any PowerApp, Microsoft Flow, or LogicApps to your custom connector, and reuse those within your entire organisation.

You could even publicly publish these if you have a service you want to expose. So if you are an ISV, this can help you get more traction on your product.

Bridging the gap

In the end it all comes down to developers of any system and language understanding the capabilities of the platforms they and their companies are using. For low code developers this means sometimes calling in the help of traditional developers. And more importantly, this also means traditional developers should learn that these low code systems can help you simplify (and thus speed up!) your development by using ready-to-roll systems and connectors available to you.

As there are over 325 connectors available, that should really speed making connections up!

Get started!

Want to explore custom connectors? Look at these resources or feel free to contact me or Daniel, we strongly believe bridging this gap between low code and traditional dev is key for succes in the future of development!

Education in a 5 feet economy

Peter Rombouts — Wed, 22 Apr 2020 08:01:00 +0000

Originally posted on SogetiLabs: https://labs.sogeti.com/education-in-a-5-feet-economy/

Our Prime Minister Mark Rutte prepared us; the 5 feet economy could be here for a while. How does that work in education? And what problems do we see around us?

Technology Leaders

I (try to) teach coworkers and clients on a regular basis, and do this mostly on-site to get the interaction going between everyone in the room, and also to get a sense of how people react to my presentation or my talk. I personally find it hard to get the same feedback while working remotely.

Mind you; we are ‘Technology Leaders‘ and are capable of doing our job and all sessions remotely. We have the tech, but do we really have the same impact? At this moment, we do not have a choice, and have to do things online.

Education Online

As a father-of-three, I see these issues also on the side of education. Kids in pre-school really learn a lot in the classroom, and also from their friends in class. Want to be engaging and keep the attention of your coworkers? Try doing that for 6 or 7-year-olds…

My belief is that, especially for children, learning from each other is key. This cannot be replaced by an online tool. Still, in these ‘5 feet times,’ you have to make concessions. Digital skills are important, so make sure your kids and your friends can work online.

I see schools struggle with Microsoft Teams, Skype, with MOOC environments and with technology in general. I see this as a fail from IT companies; We always talk about inclusion, but it is still difficult to get everyone online working together. We have done a very good job making sure ‘we’ the IT people can do everything online, but we see educators or parents in general struggling with technology.

Help the educators, help each other

Big things start small; My door is always open, and I (time permitting) am always available for a quick tip or some help. Keep in mind that using Teams or Skype or Zoom is second-nature to ‘us’, for some parents or teachers this can be challenging. By helping another parent, simply by explaining how a tool works, you already have one more person that uses the tool. Maybe you can lend out a spare laptop for someone not having those resources. Maybe you can fix their internet, or setup their environment.

Small things count, we can make a difference together.

Does language matter?

Peter Rombouts — Wed, 18 Mar 2020 07:31:53 +0000

Originally posted on SogetiLabs: https://labs.sogeti.com/does-language-matter/

In my previous blog I stated my learning goal for 2020: Learning (or getting a better understanding) of the Go language. Currently I’m studying various courses, and have to give huge credits to Todd McLeod for his excellent work on courses on this subject.

Working with what you were given

In many cases I have to work what is given. If I am helping a client running Java with Docker, I cannot simply move them to another language or framework, just because that seems a better technical solution. I have to keep in mind that people and companies have invested in a language or framework, and sometimes it seems almost a religious feud (Windows vs Linux) instead of an objective view on the subject. So in a way, a language does not matter in my work as a consultant or architect. I have my preferences, but cannot force them onto a group just for sake of my own reasons.

Let’s Go! (pun intended)

Go is a very elegant and fast language. Testing is built right into the language, as is documentation. It pretty much runs on any OS, and does that really fast.

The authors (Robert Griesemer, Rob Pike, and Ken Thompson) wanted to address criticism of other languages in use at Google, but keep their useful characteristics (Wikipedia):

static typing and run-time efficiency (like C++)
readability and usability (like Python or JavaScript)
high-performance networking and multiprocessing

In Go everything revolves around ease of programming and frankly, it is just plain fun coding in Go! I’m amazed with the speed of coding and simplicity of concurrency. Numerous IDE’s are available, making it easy to start with the language.

What language do you choose?

If you are in a greenfield environment, or want to create services that have the need for speed; think beyond your comfort zone. Experiment with languagues and frameworks. See what fits for your purpose.

In any case, take a look at Go with your team, and let me know what you think of it!

2020: Keep on learning

Peter Rombouts — Wed, 01 Jan 2020 19:32:33 +0000

This blog was originally posted under https://labs.sogeti.com/2020-keep-on-learning/

As the new year starts, many of us have New Year’s resolutions, and many of those will eventually perish within a month or two.

New Year, New Technology

I don’t have any resolutions. The only thing I try to do each year is to learn a new technique or language. Please note that new means _ new to me _ and not necessarily a brand new technique.

Why? In my day-job I focus on designing cloud native systems and architecture, and most of my ‘programming’ is done in Visio and PowerPoint. As my roots are in Software Engineering, I keep myself up-to-date by learning new languages and techniques.

For the upcoming year I’ve already made my choice. I started out with the following short-list.

Creating the short-list

The reason I chose these techniques is not random. In my work as an external examiner for the University of Applied Sciences Avans and Fontys in the Netherlands, I see the work of many students each year. They inspire me to look at specific techniques that normally do not cross my path. In my day-to-day work, the most used languages are C#, TypeScript, Java, and JavaScript languages and frameworks like Angular.

1. Scala

Scala is a general-purpose programming language providing support for functional programming and a strong static type system. Designed to be concise, many of Scala’s design decisions aimed to address criticisms of Java.

Wikipedia

Functional programming is something that I do not see often in my day-to-day job, so I was intrigued by the capabilities of this language. Also, some very fast and popular software is written in Scala. Examples are: Apache Kafka, Apache Spark and Akka.

2. Rust

Rust is a multi-paradigm system programming language focused on safety, especially safe concurrency. Rust is syntactically similar to C++, but is designed to provide better memory safety while maintaining high performance.

Wikipedia

Originally invented by Mozilla and used within Firefox and Dropbox. Rust has been the “most loved programming language” in the Stack Overflow Developer Survey every year since 2016, so that drew my attention.

3. Go

Go, also known as Golang, is a statically typed, compiled programming language designed at Google by Robert Griesemer, Rob Pike, and Ken Thompson. Go is syntactically similar to C, but with memory safety, garbage collection, structural typing, and CSP-style concurrency.

Wikipedia

Go has been around for quite some time, and has an impressive list of applications that were built with the language. Kubernetes, OpenShift, Docker and the list goes on.

Choosing my 2020 technique

For an internal project, we were looking for a tool to provides us programmability against a multitude of APIs. Instead of grabbing an off-the-shelve product, I investigated tools and frameworks that could help us build an MVP, fast and reliably. This eventually made me look into Terraform Custom Providers.

In Terraform, a Provider is the logical abstraction of an upstream API. This guide details how to build a custom provider for Terraform.

https://www.terraform.io/docs/extend/writing-custom-providers.html

Terraform will let you wrap any API, so it will enable us to wrap our ITSM tooling, Monitoring tooling and what have you not.

As Terraform and the custom providers are written in Go, that was my main reason to dive into this language. I’ve created a Github repo with an example custom provider, find it here on Github.

Keep on learning

I’m curious if any of you are also keen on ‘staying relevant’ and want to keep up with new techniques and languages. And, what techniques and languages do you try out and investigate? Please feel free to let me know, and contact me on LinkedIn or Twitter!

Infrastructure-as-Code with Pulumi

Peter Rombouts — Mon, 16 Dec 2019 14:49:36 +0000

Using a public cloud like Microsoft Azure or Amazon AWS? Chances are you’ve been using templates like ARM or CloudFormation.

Hashicorp has done a terrific job making Terraform with the ability to template any API, and deploy and manage stacks in Azure, AWS and many, many more.

I’ve been following Pulumifor a while, and their approach is a bit different than the previous stated tools.

Pulumi states their product as follows:

Modern Infrastructure as Code.

Declare cloud infrastructure using real languages. Enable developers and operators to work better together.

With ‘real’ languages they mean programming languages. At JavaScript, TypeScript, Python, Go, or any .NET language, including C#, F#, and VB are supported. I did some testing with C#, and although it is in preview, it works great:

Installing Pulumi

The install is pretty simple. This page will get you up and running in no time. I personally used Chocolatey but you can download the tool for Windows, MacOS and Linux if you want.

Getting Started with Azure

The getting started guides work great. The only thing you need (in my example for Azure) is an account. The page https://www.pulumi.com/docs/get-started/azure/ show you a step-by-step guide.

Deploying resources

I’ve picked a simple setup, having one resource group in Azure, with one Storage Account. To deploy the stack to Azure, a simple ‘pulumi up’ command fires up Pulumi. Just like Terraform, you can see what will happen if you deploy this command. In this case you see that a Stack in Pulumi is created, and a resourceGroup and storage account will be setup in Azure:

After choosing ‘yes’, you will see Pulumi actually deploying your stack. In the picture below the account has been created, and we see the output plus duration (don’t worry, the storage account has been deleted so the key not valid anymore):

Updating resources

The next step shows what happens if we update something in the stack. In this case, we have updated that the storage account only allows HTTPS traffic. The pulumi up command shows us the expected changes, and shows us it will update the storage account:

After choosing yes, the update will start, and it will display the next image. As you can see, 1 resource was updated, and 2 were unchanged:

Destroying resources

To destroy your resources, pulumi destroy will tear down everything, up until the stack in Pulumi itself:

Wait, Pulumi Stack?

Pulumi defines stacks of your deploys. To quote them on this:

Every Pulumi program is deployed to a stack. A stack is an isolated, independently configurable instance of a Pulumi program. Stacks are commonly used to denote different phases of development (such as development, staging and production) or feature branches (such as feature-x-dev, jane-feature-x-dev).

So, you can look at these like your set of resources. You can group them under Projects so it can be easier to manage. The webinterface of Pulumi is clean, and you can inspect all of your stacks and settings there:

Conclusion

This short blog is just to show you that there is more than the standard way of working with the cloud. Obviously you can use various APIs, SDKs or templates, but these sometimes can be a bit overwhelming for developers. Pulumi lets you choose your language, and makes sure your developers do not need to learn another language and still use the modern infrastructure-as-code techniques!

Sync your Outlook 365 calendar to Google Calendar with LogicApps

Peter Rombouts — Wed, 04 Dec 2019 18:18:42 +0000

Sometimes it is hard to keep your spouse up-to-date on all the sessions and events work-related. Most of the times I’ve been able to duplicate calendar items that are relevant for my ‘better half’.

Obviously manual work makes me sad and I forget to sync some events causing a collision in the calendars. If, like me, you have kids you’ll understand this results in hoping your babysitter can make time because you forgot to add some items in your calendar manually.

LogicApps to the rescue

LogicApps (and also Microsoft Flow) have connectors to both Office 365 and Google. Seems like a sure bet to get my sync going!

Setup Google Calendar

First of all, I’ve created a new calendar in Google. I want to separate my personal items from work, and want the ability to enable/disable the view. A new calendar is the easiest way of doing this. Also if you share your Work calendar, the people you share your calendars with, can easily differentiate between what is work and private. Google has an excellent manual, click on this link to follow the setup of your new calendar.

Setup LogicApp

First, you will have to setup your LogicApp. If you don’t know how, follow this tutorial from Microsoft.

The first step is to add the trigger for Outlook. I’ve used the following trigger:

This will allow me to react on added, updated and deleted events. I check this every hour, and track upcoming events 300 days in the future. Because I don’t really care about days in the past, I set that to 5 days. Adjust these to facilitate your own needs. I’ve set the interval to 15 minutes, which works for me. If you have a long interval, you could get 429 errors from Google because you’re pushing too many changes to the Calendar at once.

Handling events

The LogicApp receives adds, updates and deletions. In my very simple sync, I add the Outlook ID to the description field of the event. Because I’m not interested (or lazy) in the actual updated info, I’ll just delete all events with the incoming ID, and create the event as new:

So at this point any item created from Outlook is deleted.

Insert Google Calendar items

Because we deleted the existing calendar item, we do not care if this is a update or create. The only events we can ignore are the deleted ones as we already deleted the calendar items. We use a switch for this:

As you can see, the default action is to create an event, and in the deleted case we simply do nothing.

The events of Outlook differ a tiny bit from the events in Google. If we zoom in on the event creation, please notice the Z behind the datetimes:

If you do not add those, Google will not accept the JSON as it will give an error on invalid data. Also seen here is the Outlook ID I use to identify the events coming in from Outlook 365.

Finishing up

At this moment you can save the logic app and you are done! You can obviously do more fancypants stuff like adding more info to the events, or handling CRUD more elegantly than me, but the basics are there for you to play with!

Below is an example of my personal calendars, and the newly synced black work calendar:

Costs

But wait; This stuff costs money right? Absolutely. But if I look at the forecast in Azure, I see the following forecast:

It is safe to say that for less than one euro, you get alot of bang per buck!

Conclusion

It is easy to sync your calendar using LogicApps or Flow. Using the connectors the possibilities are virtually limitless! If you have any manual operations you want to automate, look at the connectors and create your own app for automation. Have any other ideas, please share them with the community!

For the code of this app, click here to view the gist.

Software Maintainability in the Cloud Era

Peter Rombouts — Mon, 14 Oct 2019 07:33:17 +0000

The shift to cloud, and with that, to PaaS services or low code alternatives like LogicApps push the actual code developers see and use to the background.

There is an ISO standard on software quality, and the maintenance best practices are well written and explained in the book Building Maintainable Software. Within low code systems, applying these guidelines can be less obvious and it can be a difficult task automating and testing the quality of your code with tools like SonarQube.

Should we even worry about the underlying code? Absolutely. The principles still adhere and creating a spaghetti of your low code systems can cause major issues on maintenance or adding new features.

Let’s focus on three points of the maintainability guidelines:

Write code once
Couple architecture components loosely
Automate development pipeline and tests

In no way these are the most important items, but for this example an easy entry into the low code space.

1. Write code once

Just like any other audit of software, you still can avoid writing duplicate code. In a platform like LogicApps it can be easy to repeat a custom call to something like a custom HTTP API.

In traditional languages like C#, you have many options to reuse your code. You could create a library, or you can create a package and make it available via NuGet. Within low code systems, these same packaging mechanisms not always exist.

Taking the example of LogicApps, the solution could be to create Custom Connectors. These will wrap your custom API calls into a reusable component you can share within your organisation, or even outside.

2. Couple architecture components loosely

If your components are tightly coupled, it can be troublesome to replace or refactor your components. The impact will be on each and every other component that has a high coupling with your part.

Again taking the LogicApps for example, let’s state our LogicApp calls another component directly using HTTP. The Azure Portal gives you this out of the box, letting you call Azure Functions directly from your LogicApp.

This goes agains the principle of loosely coupling. The reason is that your call is directly bound to that function, so changing the interface or location of your function, impacts the LogicApp directly. In this case, it would simply break and stop working.

To solve this problem, a simple solution is to decouple the LogicApp and the Azure Function using a queueing mechanism. This way, the message to the Azure Function is put on a queue by the LogicApp, and the Azure Function listens on a queue. Now, if the developer of the Azure Function changes location or even use another platform, there is no need to change the LogicApp.

Obviously this would require you to make an agreement on the contents of the messages on the queue.

3. Automate development pipeline and test

Automating your CICD pipelines allows you to more easily build, test and deploy your code. In case of a language like C# or Java, you can easily run tests, build your code, and create packages or deployments. A tool like Azure DevOps can combine these steps and lets you create a wealth of quality gates, checks and processes to guide your team.

When using low code platforms, it can be cumbersome to get the code into your version control systems. Nevertheless, many of those platforms do give you the tooling. LogiApps for example has template creators, and systems like OutSystems have their own CICD ecosystem.

In the end the automation allows you to more easily add steps to your CICD process, and allow you to deploy more frequently without any hassle. The addition of automated testing will absolutely be beneficial to the overall quality of your product.

Concluding

Treat your low code just like you would any other codebase. Almost all guidelines of mainainability can be mapped to your product. Some can may require a bit more investment, but in the end I truly believe it will help building a maintainable and high-quality (low) code base.

If you want to expand on the process part of high-quality software, please also take a look at the follow-up book of this series: Building Software Teams

Migrating from Jira to Azure DevOps

Peter Rombouts — Fri, 09 Aug 2019 08:27:32 +0000

As far as I’m concerned, a team can use any tool they want to track their work items and planning. But, my personal favourite is Azure DevOps. Currently I’m helping some teams moving their Jira projects to DevOps and this blog explains how you can do this too!

Uniformity and standards

For several reasons it can be frustrating if teams use multiple tools for the same purpose. For sake of uniformity, I believe it is best to have standards within your company. In the case of my current customer, that standard is Azure DevOps.

Obviously you want to migrate your work from the old to the new platform. Azure DevOps has the ability to use APIs to ingest data, and you can export Jira as well. To do this manually and figure out all the moving parts, you will have quite a challenge.

Enter Solidify

I was pointed to a really great tool by the team of Solidify, the Jira to Azure DevOps work item migration tool.

The Jira to Azure DevOps work item migration tool lets you export data from Jira and import it as work items in Azure DevOps or Microsoft Team Foundation Server

This sounds exactly like what we need! Because every project is different, it can be tricky setting up all the mapping of statuses, and figuring out how to connect to the correct instances.

The documentation of the tool is great, but this blog will give a step-by-step approach on how to do a migration.

Getting the tool

The Github link in the beginning of this post is the starting point for all documentation of the tool. To download the release directly, point your browser to solidify/jira-azuredevops-migrator/releases.

Preparing for your Jira export

The Github repo contains samples at this location. Worth noting is that the setup of the configuration file is essential. See the first few lines of the file below:

"source-project": "MyJiraProject", "target-project": "DevOps Project Name", "query": "project = 'MyJiraProject'", "workspace": "C:\\Temp\\JiraExport\\", "epic-link-field": "Epic Link", "sprint-field": "Sprint", "download-options": 7, "batch-size": 20, "log-level": "Info", "attachment-folder": "Attachments", "user-mapping-file": "", "base-area-path": "", "base-iteration-path": "", "ignore-failed-links": true, "process-template": "Agile",

As you can see, the source and target fields correspond with your data in Jira and DevOps respectively. The process-template field corresponds to what you have setup in DevOps, like CMMI, Scrum or Agile (or Basic).

Next to setup, are the mappings. As you might know, some tasks, statuses or other properties differ in Jira and DevOps. Let’s say you have a Task in Jira with status Review. You might not have the same status in DevOps, so you want to map that to another status type. The same goes for types like ‘User Story’ or ‘Epic’:

"type": [{ "source": "Feature", "target": "Feature" }, { "source": "Epic", "target": "Epic" }, { "source": "Story", "target": "User Story" }, { "source": "Bug", "target": "Bug" }, { "source": "Task", "target": "User Story" }, { "source": "Sub-task", "target": "Task" }]

Within the configuration file, you should specify all these mappings. If you fail to add one, the export tool will generate an error, so you can add the missing mapping and re-export Jira.

Running the Jira Export tool

If the config file is setup, you can run the tool using the following command:

jira-export.exe -u p.rombouts@yourdomain.com -p api-token-here --url https://yourinstance.atlassian.net --config config.json

As you can see I’ve setup the command using the API token. Please refer to this page on how to get an API token for Jira. In theory a password should work, but it did not work in my case.

Inspecting the export results

When the tool runs, you will see results directly in your CLI interface:

$ jira-export.exe -u username -p apitoken --url https://something.atlassian.net --config config.json Jira Exporter v2.2.25 Copyright (c) Solidify Wow, there's a newer release out (v2.2.31)! We recommend downloading it for latest features and fixes. [I][10:12:11] Connecting to Jira... [I][10:12:11] Retrieving Jira fields... [I][10:12:12] Retrieving Jira link types... [I][10:12:12] Export started. Exporting 705 items. [I][10:12:13] Initializing Jira field mapping... [I][10:12:14] Processing 1/705 - 'Item-790'. [W][10:12:14] Could not find user 'someuser1@yourdomain.com' identity in user map. Using original identity 'someuser1@yourdomain.com'. [W][10:12:15] Could not find user 'someuser2@yourdomain.com' identity in user map. Using original identity 'someuser2@yourdomain.com'. [I][10:12:15] Processing 2/705 - 'Item-789'.

As seen in the above start of the export, some users could not be found in the mapping. This is done on purpose by me; I want the exact same users from Jira to be used in Azure DevOps. Optionally you could create a users.txt file, and map users in Jira to different users in Azure DevOps. If any fields or mappings are missing, you will also see this in your console.

Bonus : In the export directory specified in the config tool, the output is also saved to a file called something like jira-export-log-190809-101211.txt. So you do not need to save the output yourself and just let it roll!. The export can take quite some time, so sit back and grab a coffee.

Running the Azure DevOps Import tool

If the previous step completed without any mishaps, and all looks OK in the log file, you can import your Jira exported data into Azure DevOps.

Again using the same config file, you can run the following command:

wi-import --token personal-access-token-here --url https://yourorganisation.visualstudio.com --config config.json --force

As you can see I’ve setup the command using the Personal Access Token. Please refer to this page on how to get one for Azure DevOps.

Again this tool will provide you with verbose information, and will also dump a logfile to the same folder as where the Jira export resided.

If everything went OK; you are now good to go in Azure DevOps!

Finalizing and cleaning up

Process wise it is smart to plan this ahead, with your team. You should have a cut-over date, and make sure that access to the old Jira project is read-only, or turned off, after you’re done exporting/importing. You do not want to maintain two boards for a team!

Conclusion

Hope you find this manual useful, the tool helped me out a great deal. The documentation is excellent and it is actively maintained by the fabulous team of Solidify. Can’t thank you guys enough!

Scanning Blob storage for viruses with Azure Functions and Docker

Peter Rombouts — Mon, 15 Apr 2019 19:00:00 +0000

While working on a side project, someone asked me how to scan for viruses in a cloud native and serverless landscape. That made me think about a project I did a couple of years back. During that project we used ClamAV that was installed on a VM. We scanned files that end-users uploaded within an ASP.NET application, and everything was hosted on datacenter VMs somewhere in the Netherlands.

ClamAV® is an open source antivirus engine for detecting trojans, viruses, malware & other malicious threats.

In this blogpost I will show a proof of concept using a Docker image and Azure Functions to create a simple automated virusscanner for Azure Blob storage.

Setting up the Docker image

First of all some prerequisites. In this demo I’m running Docker for Windows on my Desktop, and coding in Visual Studio 2019 with local Azure Functions. I’ve already setup a Azure Storage account that I can use as a trigger in my Functions.

First, let’s find a Docker image with ClamAV installed.

There are several to be found on DockerHub, I’ve pulled mkodockx/docker-clamav and it works seamlessly in my demo. To run this, simply execute the following command:

docker run -d -p 127.0.0.1:3310:3310 mk0x/docker-clamav:alpine

It should be running and listening on the default port on localhost.

Creating the Azure Function

The second step is to create a project for our Azure Function:

We’re using the blob triggered function, because we want to scan all new blobs in the location ‘upload’.

Now, for scanning, the following NuGet package for nClam should be installed:

Install-Package nClam -Version 4.0.1

The actual code for scanning is pretty simple. Shown below is the code that grabs the file, uploads it to the Docker container and retrieves the status of the scan:

using System.IO; using System.Linq; using Microsoft.Azure.WebJobs; using Microsoft.Extensions.Logging; using nClam; namespace VirusScanFunction { public static class ScanBlobs { static readonly string serverName = "localhost"; static readonly int serverPort = 3310; [FunctionName("ScanBlobs")] public static void Run([BlobTrigger("upload/{name}", Connection = "AzureWebJobsStorage")]Stream myBlob, string name, ILogger log) { // Create client var clam = new ClamClient(serverName, serverPort); // Scanning for viruses... var scanResult = clam.SendAndScanFileAsync(myBlob).Result; switch (scanResult.Result) { case ClamScanResults.Clean: log.LogInformation("The file is clean!"); break; case ClamScanResults.VirusDetected: log.LogInformation("Virus Found!"); log.LogInformation("Virus name: {0}", scanResult.InfectedFiles.First().VirusName); break; case ClamScanResults.Error: log.LogInformation("Error scanning file: {0}", scanResult.RawResult); break; } } } }

That is basically it! You just created your own local virusscanner for your blob storage!

Recap and considerations

Obviously this is a PoC and runs fine ‘on my ~~machine~~ container’. For enterprise environments you should care about what container image to use, how to handle viruses found, and what to do to mitigate problems. You obviously want to warn, and block, the person who uploaded the virus in the first place.

Conclusion

Serverless and containers allow you to setup really cool concepts, and make you think outside the box. Years ago we had to install ClamAV on a VM, where now we can use containers, spin them up, scale them, and dispose when we are done. Think of the possibilities!

LogicApps monitoring with Log Analytics

Peter Rombouts — Mon, 01 Apr 2019 20:56:00 +0000

Azure LogicApps are incredibly powerful and I believe it should be in every developers toolbox. Enterprise integrations, ready-to-use connectors and all of this with almost no code and serverless!

Alas monitoring and debugging are more complex. In this blog I will explain how to add LogicApps to Log Analytics, and even track custom properties for filtering and alerting. Keep on reading for the exciting stuff!

Create a LogicApp

Let’s start out with a very simple LogicApp. This app will receive a HTTP request with an array of data, and for each item, send an email:

Standard logging

The history of runs is shown below. You can drill down and see the actual results of each call, and see what part took how long and more details.

Enter the Log Analytics workspace

Standard logging seems fine, but what if you want to integrate this LogicApp and have alerting setup? First of all, setup a Log Analytics workspace. When you created the Log Analytics Workspace, make sure your LogicApp sends logs to it. You can simply go to the Logs section of your LogicApp:

As you can see, I already set mine up to the existing one. When that is done, you can click on the Log Analytics workspace, which brings up the overview blade of your workspace:

Clicking on ‘View solutions’ on the ‘Configure monitoring solutions’ will open up the settings. Click on ‘Add’ and search for Logic Apps Management. Please note that this is in preview at the moment of writing:

If you select Create and choose the correct settings, it will show the runs and logging in the Workspace Summary section of Log Analytics:

As you can see, we see a nice chart with basic info. Drill down on that and you will see all LogicApps reporting to this workspace. Note that I’ve only configured one:

Detailed Logging and resubmit LogicApps

If we zoom in even further we can rerun several LogicApps and view more detailed information:

Here we can click on Logs and zoom in even further, or we can resubmit multiple LogicApps.

Tracking custom properties

Let’s say we have many, many events coming in, and we want to track by CustomerID. For the LogicApp, we have to define tracked properties. For this go back to the LogicApp, enter Code view, and add the properties you want to track:

"runAfter": {}, "trackedProperties": { "CustomerId": "@{items('For\_each')?['data']?['customerid']}", "Flow": "Email", "Receiver": "SomeReceiver", "Subject": "@{items('For\_each')['subject']}" }

To explain the piece of code above; I’ve added the ‘trackedProperties’ in the forEach loop, and this will send these custom properties to Log Analytics. Obviously these properties correspond to the event we got in the HTTP Request.

After running this app now, we can see tracked properties in Log Analytics! If we click on the ‘ View ‘ button, it will bring up the actual properties:

Alerts and filters

Now for the advanced scenario, we want to create a query and search on the properties:

In the above sample, you can see that a simple filter on schema will bring up all tracked properties and on the right, you can see a simple query to retrieve all events for ‘ Customer 1 ‘. That is basically it! Now you can click on ‘ Create Alert Rule ‘ and do your magic there, and create alerts using these tracked properties!

Conclusion

Setting up the LogicApps to communicate with Log Analytics is a breeze (yes still preview) with the Logic Apps Management module. Tracking properties is a bit of manual labour, because you have to enter code view, but still no worries there. Do look out; if you add the tracked properties in the wrong portion of the JSON, you will not see anything working. And also don’t forget that it can take some time before all logs are updates. Happy low-coding!

Cost optimization in Azure

Peter Rombouts — Fri, 15 Feb 2019 15:08:08 +0000

In my previous post I showed how easy it is to start or deallocate machines using C# and Azure Functions.

Azure VirtualMachines can be handy but very costly if you forget to turn them off∗ and leaving them running for 24/7.

But, what if you have dozens of existing VMs and are looking for a quick way to schedule the on/off times for those machines to save money?

In this post I will show a very simple script to use with Azure Automation that enables you to get this up and running, literally within the hour.

Note: I would advise to use Azure DevTest labs if you have development virtual machines. That will give you the opportunity to reuse, claim or share VMs within a group of users.

* In this blogpost I will use the word Stop in code and comments, but in fact that will Deallocate the machines, and will stop incurring costs.

Calculating costs

First of all, lets take a quick look at how much you can actually save with the instructions in this blogpost. If we take a D4s v3 machine and let it run for 24/7, it will cost approximately 260 euros a month (in West Europe). If you have a team of 8 developers, that adds up to more than 2000 euros a month.

For sake of this blogpost, let us state the actual working hours are (monday to friday) from 7 AM until 7 PM. This brings the hours down to 12/5. The running costs of one machine will be lowered to approximately 92 euros a month. For a team of 8 developers, this will save over 1300 euros a month , each month, with the use of a simple script.

Azure Automation

I will not explain how to setup Azure Automation, as that is pretty simple to setup using the Azure Portal. I’ll jump right into importing and creating runbooks. We start in the Azure Automation blade in the portal:

Click on ‘Runbooks’ in the left pane, and you’ll see some predefined runbooks added by the system.

Please download my PowerShell script for importing here

After downloading my script, you can click on ‘Import runbook’ in the menu bar of the blade.

After importing, the runbook will have the status ‘New’:

To start using the runbook, click on the ‘Stop-Start-AzureVM-By-Tag’ runbook. In the next screen, click on ‘Edit’:

This will open up the actual runbook script, and show all the PowerShell code. In here you can adjust the code to your liking if you want to add logic or change parameters:

Now, click on ‘Publish’ and you can run your first runbook!

Tagging your VMs

Obviously you do not want to manually click to start or stop the machines. In my simple code example, I’ve filtered on a specific tag name and tag value. Tags can be set programmatically or manually. For playing around with this script, simply go to a VM, and make sure the tags are set up like this:

The tagname is ‘ ScheduledOnOff ‘ and the value is ‘ Yes ‘.

Scheduling your runbook

At this moment the runbook is not automated. There is no schedule attached, but you could manually trigger the runbook to start or stop your machines.

To create a schedule, go back to the runbook, and select ‘Schedules’:

Creating a new schedule is easy; I’ve created one schedule to start at 07.00 at weekdays:

The final step is to setup the parameters. In my predefined script, using the default parameters, we only have to set ‘Start’ in the action field:

The same setup should be followed to deallocate the VMs, creating a schedule with the ‘Stop’ action at the preferred time of day.

Conclusion

This blog explains how to save resources and reduce costs in a very quick and effective way. This can be seen as a patch or temporary solution, because if some user deletes or changes the tags, the script will not work for that specific VM anymore.

Nonetheless it is a very handy, quickly implemented solution to fix your long-running-always-on dev or test machines!

Managing VirtualMachines in Azure with C#

Peter Rombouts — Thu, 24 Jan 2019 09:29:57 +0000

Managing VirtualMachines is often done using Azure Automation or PowerShell. But what if you want to do this using C#, in an Azure Function? This post shows a very simple example on how to implement this in a .Net Core 2 Azure Function.

Setting up the project

To get started, simply create an Azure Function project in Visual Studio:

Setting up the function

For this example I’m going to create a Function with HTTP trigger that deallocates all VMs in a resource group.

To get started, create a new function called ‘DeallocateVMs’:

I’ve chosen the ‘HTTP’ trigger because this is the easiest to run and debug on your local machine.

Setting up the packages

First of all, install the next NuGet packages:

Install-Package Microsoft.Azure.Management.Fluent Install-Package Microsoft.Azure.Management.Compute.Fluent

This allows us to make use of the Azure libraries we need to manage VMs.

Boilerplate code

The library needs to connect to Azure and have rights in the resource group you want to manage. For this, we need some boilerplate code to get the AzureCredentials :

internal static AzureCredentials GetCredentials() { var clientId = "Enter ClientId Here"; var clientSecret = "Enter ClientSecret Here"; var tenantId = "Enter TenantId Here"; return SdkContext.AzureCredentialsFactory.FromServicePrincipal(clientId, clientSecret, tenantId, AzureEnvironment.AzureGlobalCloud); }

For more info on creating Service Principals, please check out this link at Microsoft

After getting the credentials, the client can be configured:

var azure = Azure .Configure() .WithLogLevel(HttpLoggingDelegatingHandler.Level.Basic) .Authenticate(credentials) .WithDefaultSubscription();

Getting all VMs in a resource group

The IAzure interface allows us to list VMs, get information, and a lot more. Please check the recap of the blogpost for links to documentation.

In this example we want to deallocate all VMs in a specific resource group. I’ve skipped all checks and exception handling for sake of readability:

var resourcegroup = "your-resourcegroup-name-here"; foreach (var virtualMachine in await azure.VirtualMachines.ListByResourceGroupAsync(resourcegroup)) { log.LogInformation($"Deallocating VM {virtualMachine.Name}..."); await virtualMachine.DeallocateAsync(); log.LogInformation($"Deallocated VM {virtualMachine.Name}"); }

That is basically it! This function will now iterate thru all virtualmachines in the resource group and deallocate them

Recap

This example does not take into account what happens when you have very large numbers of virtual machines. The code also does not take into account what the current PowerState of the machine is. In a more advanced scenario, you should check the PowerState and determine what to do. The easiest way is to check if the machine is actually running:

if (virtualMachine.PowerState == PowerState.Running) { // Do something }

I’ve used a service principal with clientid and secrets, and set the rights on the managed resource groups. This is a very basic setup to show how to manage the VMs. There are alot of ways to implement security, and maybe your scenario has client certificates or other authentication options. This post on Github shows many scenarios on authenticating with this library.

Links

For this example, I’ve used the namespaces Microsoft.Azure.Management.Compute.Fluent and Microsoft.Azure.Management.Fluent Namespace. There are a lot of exampleson Github using the two NuGet packages.