DEV Community: MertSenel

Why Allowing Engineers to 'Fail Safely' Is Crucial for High-Quality Software Development

MertSenel — Sun, 22 Oct 2023 10:48:06 +0000

In the ever-evolving landscape of software development, there's been increasing chatter about how the process of building, testing, and deploying software has ostensibly become easier. With a plethora of new toolsets available to software engineers and adjacent tech professionals, like quality engineers, cybersecurity experts, and DevOps engineers, one might believe the journey has become smoother. However, this perspective tends to oversimplify the inherent complexities.

Yes, automation, generative AI, and specialized tools have streamlined many mundane tasks. Yet, the expectations from engineers have simultaneously risen, particularly in the domain of business context. Modern engineers are more entwined with their products and services than ever before, directly influencing the financial bottom lines of businesses, partners, and customers. Those who excel in this environment are not just technically proficient but also possess a keen understanding of business vision, strategy, and principles. Such a holistic perspective aids in decision-making, especially in roles that require frequent determinations.

However, as distributed systems expand and integration points multiply, the cognitive load on engineers intensifies. It's crucial, now more than ever, for engineers to operate within a well-structured and efficient environment, one that aids in managing this cognitive overload and ensures optimal outcomes.

The Philosophy of "Failing Safely"

A term that's gained traction in the tech community is the concept of "failing safely." I recently attended a software testing conference, and one statement deeply resonated with me: "There's no 100% bug-free software or system. Striving for perfection would only allow our competitors to outpace us in the market.This sentiment underscores the inherent risks every time we ship software.

But how can we mitigate these risks? The answer lies not just in the post-development phase but primarily in the pre-deployment testing phase. This principle holds for both software developers and the architects behind the infrastructure of these intricate distributed systems.

Key Components of a Safe Development Environment

Local Testing Capabilities: The foundation of any development process is the ability to perform local testing. Engineers should have the tools and environments that mirror real-world scenarios as closely as possible. By simulating external integrations, databases, and other dependencies locally, developers can identify and fix issues before they reach a shared environment. Utilizing containerization technologies can further standardize this process, ensuring that all team members work within consistent parameters, reducing the "it works on my machine" phenomenon.

Disposable Environments: The cloud age has brought about the possibility of spinning up environments on-demand and tearing them down when no longer needed. These temporary environments are invaluable for testing complex features, especially in distributed systems where interactions between components can be unpredictable. By defining infrastructure as code, these environments can be version-controlled, ensuring repeatability and consistency across various stages of development.

Sandbox or Playground Areas: Offering engineers a safe, isolated space to experiment and learn is essential for innovation. These sandboxes, separate from production and even development environments, allow for risk-taking without the fear of negative consequences. Such areas are not just for testing new features but also for training, upskilling, and trying out new technologies or methodologies.

Automated Rollbacks: In the fast-paced world of continuous deployment, the ability to quickly revert changes is paramount. Automated rollback mechanisms ensure that if a newly deployed feature causes issues, systems can instantly return to a previously stable state. This reduces downtime and user impact, ensuring that services remain reliable even when deployments don't go as planned.

Monitoring and Observability: Beyond traditional monitoring, observability provides insights into the internal state of systems based on external outputs. It's not just about knowing when something goes wrong, but understanding why. By integrating comprehensive observability tools, engineers can get a granular view of system behavior, facilitating quicker debugging and more informed decision-making.

Blameless Post-Mortems: Mistakes are inevitable. What matters is how teams respond. Cultivating a blame-free culture where the focus is on learning and improving, rather than pointing fingers, is essential. Blameless post-mortems encourage open discussions about what went wrong and how to prevent similar issues in the future, fostering a culture of continuous improvement.

Feature Flags: These are powerful tools that allow developers to release new functionalities in a controlled manner. By toggling features on or off, they can be tested in live environments with a subset of users, gathering feedback and ensuring stability before a full-scale rollout. This granular control reduces risks associated with deployments and allows for more iterative development.

Encourage Pair Programming: Two heads are often better than one. Pair programming promotes knowledge sharing, reduces coding errors, and fosters a collaborative team culture. By having two developers work on a piece of code together, they can brainstorm solutions, review each other's code in real-time, and ensure that the best possible solution is achieved.

In conclusion, creating a nurturing development environment is about more than just tools and processes; it's about fostering a culture where engineers feel empowered to innovate, learn from mistakes, and consistently deliver high-quality software. By focusing on these key components, organizations set the stage for excellence, driving growth, and ensuring long-term success.

Stages of Platform Engineering and State of DevOps

MertSenel — Wed, 18 Jan 2023 11:19:07 +0000

Platform Engineering and Different Stages of DevOps Maturity

Platform Engineering is term you might have heard of but not yet got to know what it exactly is. Platform's are a set of infrastructure that takes care of compute, persistent storage, networking and utilities such as monitoring, alerting and observability. This is a convenient way of boosting velocity in going to market for a service that is being developed. One of the popular recipe is having a compute platform at core such as Kubernetes and complementing it with cloud PaaS offering to provide things such as, messaging, storage, database or caching or similar infrastructure. If you also have pre modelled designs for high availability, geographical redundancy, security you should be having a streamlined process for launching services to production in small amount of time.

Today I wish to talk about some maturity levels in terms of DevOps automation that makes up your capability to onboard and launch services on your Platform. You might have a great platform but often times the tooling and automation behind your platform defines to ease up adoption for new services and developers using your platform. Is your platform team is required to aid developers and squads to onboard themselves and their services on your platform or are you in a state to offer "paved roads" to them that they can easily make use of and be happy with the end result. As most things in DevOps the answer is not binary and you can assess and see which stage you are at, right now. Where are you today, also should define the next milestone you should aim for, and finally achieve a cutting edge platform capability for your team and your stake holders. Without further due lets get into the different stages of maturity and what they exactly means.

Stage 1: Lack of Capability

This is a stage where your Platform team doesn't know how to solve a problem. For example a delivery squad may come to you with requirement such as, they want to have a containerized application, with need of a NoSQL database and a Redis cache. Your team at this moment, don't have a reference on how to continuously integrate this solution, how to consistently, build, lint, test, security scan, package and distribute such software. Then there is the problem of having a highly available, redundant and secure infrastructure that you can monitor, alert and observe that would allow you to operate this piece of software in a production environment.

This state is your starting point and you may be at this stage, this is the point where you may decide on some long term solutions. If you expect a lot more of similar problems down the line, you may decide on building a platform so you can have reusable solution to these requirements for workloads of same or similar nature.

Stage 2: Custom Solutions

You have now started building a solution based on the request you have received. You started developing infrastructure as code, pipeline as code, configuration as code to package and deploy the service. You may parametrize the artifacts you build to be able to accommodate different Software development lifecycle environments such as development, qa, uat, staging, production with different levels of SKUs and redundancy levels. However at this stage whatever you built is most likely bespoke and it would still be labor intensive to scale this same solution for other services.

The best solution you have at this stage is to duplicate the parametrized artifacts and alter them to fit for the next service you need to onboard and host on your platform, but it still requires a good amount of effort and it needs your platform team's direct involvement to enable the delivery squad to develop and launch their service.

Stage 3: Reusable & Central Artifacts

At this stage, you have moved all of your parametrized artifacts of pipeline, infrastructure as code, configuration as code monitoring artifacts into centralized repositories. The reusable artifacts are developed in a manner that they are highly parametrized and considered as generic modules that can accommodate a wide range of scenarios.

As a platform team you provide and maintain reference architectures that showcases a recommended way of consuming the generic modules you have developed and this provides a streamlined way of onboarding and developing a service that would by design be compliant and secure that has the serviceability and supportability requirements fulfilled from get go. This is called a "paved road", it would be the road that'd be taking has benefits in terms of being the road to production with least of resistance. It'd be encouraged and desired to take by everyone as it'd have better support from the platform team and provide convenience to developers while developing new services.

Stage 4: Automated Bootstrapping

As patterns emerges, the platform team should be able to spot certain use cases on how their platform it getting used. They may even define some baseline resources, common denominators they'd consider as a bare minimum requirements for a service. At this stage platform team would create some high level automation that leverages the distributed automation workflows they have already built in the previous stages to create high level automation workflows that'd make the onboarding and progression to production much easier and streamlined for new services.

This top level automation should can be more opinionated and should able to configured by small set of parameters such as project keyword, environments needed and small set of selections to be made by the consumer. At this stage the consumption of the automation is so easy that the consumer can be both the platform team that provides and maintains this automation workflow or the consumer can be directly the delivery squad that would be developing the service they wish to onboard and launch to production on the platform. This stage provides better velocity, developer experience as the involvement required from the platform specialist is lowered.

Stage 5: Environment as a Service

At this stage the platform team incorporates a platform orchestration tool to abstract the automation one more level and provide a way to orchestrate even more automation in one single flow. By using specialized tools, they allow to provision and maintain environments with minimal effort as orchestrator acts as a top level actor that initiates and governs the environments generated by this tool.

The environments generated, although should be fit for purpose for majority of the cases, the platform team would still be in consultant capacity to provide the "last mile engineering" to fill in the gaps and make the solutions provided satisfying all of the requirements of services if they need this assistance.

The environments generated at this stage are all adhering to "paved road" principles that are secure, compliant and reliable by design. The platform orchestrator provides a process to provision and manage the environments created with the Day 2 activities such as maintaining and updating in a high scale.

Stage 6: Internal Developer Portal

Internal developer portals, are internally available web applications that provides a self-service portal for developers to consume the "paved roads" developed by the platform team. At this stage as the consumption and governance shifts towards the delivery squads the platform and it's offering also makes a shift in maturity and model.

At this stage, the platform and it's features are considered as a product. The platform team being the custodians of the platform would still be required to educate delivery squads and provide consultancy to delivery squads to consume their offerings but their main duty is to maintain and update their platform, as a product.

Internal Developer Portal, acts as an "App Store" for infrastructure and supporting tooling for a service. The consumers are not only be able to provision resources required for their services but also be able to view the things that they own, manage and perform administrative tasks on them via using the developer portal. This stage is considered as the pinnacle of the automation & DevOps maturity that you can have for your platform and it can have an excellence level as much as you are willing to put efforts into it. Just reaching this stage as a minimum viable product is a pretty significant achievement, but from here onwards sky would be the limit.

Conclusion

In DevOps, the iterative value delivery is one of the key factors. The important thing is to have a direction to follow and align your efforts to reach the state you are aiming to reach. I have tried to give you an overview of different maturity levels for DevOps and automation capabilities, so you can asses what your organisations current level is and what target state you can aim, for your next sprint of efforts. I hope this you find this article, helpful in understanding your current state and strategising your ideal state in the upcoming future.

SSL Certificate Expiry Checker

MertSenel — Sat, 17 Dec 2022 16:45:14 +0000

Track Your SSL Certificate Expiry Dates via CI/CD Pipelines

Custom HTML Report Sample

Code Repository: https://github.com/MertSenel/ssl-certificate-expiry-checker

Background

Operation teams are responsible for rotating SSL certificates of the services they are responsible for which have HTTPS endpoints.
Although it's better to automate the full rotation process, this may not always possible. Regardless of the rotation process a regular check can help with keeping on top of overall SSL certificate health, expiry being the most important metric. We also know that, as of 2020 most certificate authorities limit their maximum validity period for newly issues certificates to 1 year. This adds a big burden to teams which are responsible for multitude of domain names and SSL certificates issued for relevant subject names.

Solution Design

The main focus of this project is to perform a regular check to your HTTPS endpoints and check the expiry date of your SSL certificates used by that endpoint. The full certificate object is being collected, in this implementation I am only reporting based on expiry dates but other checks can be added as well. The reason to use CI/CD pipelines hence agents is to allow operation teams to run these tests with runners/agents that has the network line of sight to both their public facing and internal origin endpoints.

This tool will do a TCP based retrieval for the SSL certificate of remote endpoint and holds generates a report in JSON format. Afterwards this JSON report is used to generates a custom HTML report for publishing and to be used as a body of an HTML based notification. The pipelines will also generate a test report in JUnitXml for easier reporting of pipeline run status and reporting.

The solution has a pipeline/workflow sample for both Azure DevOps Pipelines and Github Actions based on your choice of CI/CD tooling. Both has different capabilities but both are popular and valid choices.

Customizable within the utility scripts, the current alerting thresholds and color codes are based on table below.

Alert Threshold in Days	Severity
90	Low
60	Medium
30	High

Azure DevOps Pipelines

Cron Schedule, allowing for periodic checks, reporting and alerting
HTML Viewer extension is used to view the HTML report as a pipeline result
Pester Test results are reported in the pipeline for seeing if any of the endpoints checked has an SSL certificate with expiry date within alerting periods.
The test results are also allows us to report on pipeline's result status. An easy way for getting notified from the pipeline can be achieved via subscribing to the pipeline's results in various ways Azure DevOps has native integration such as email, or Microsoft Teams Azure DevOps bot.
Alternative e-mail based or webhook based notifications can be added via the utility script Send-Notifications.ps1.
Azure DevOps project dashboard widgets such as test results trends or pipeline build result widget can be used to track status at a glance inside Azure DevOps.

Sample Azure DevOps Pipeline Run Screenshots

Sample Azure DevOps Pipeline Run

Sample Azure DevOps Pipeline Run Test Results

Sample Azure DevOps Pipeline Run Custom HTML Report

Sample Azure DevOps Dashboard Sample

GitHub Actions

Sample GitHub Actions Workflow Run Screenshots

Cron Schedule, allowing for periodic checks, reporting and alerting
As HTML Viewer extension is not available for GitHub Actions, publishing report to GitHub Pages has been used as an alternative. GitHub Pages requires either a public repository or a paid GitHub Account, if wished to be used in a private repository.
Pester Test results are reported in the pipeline for seeing if any of the endpoints checked has an SSL certificate with expiry date within alerting periods.
The test results are also allows us to report on pipeline's result status. An easy way for getting notified from the pipeline can be achieved via subscribing to the pipeline's results in various ways Azure DevOps has native integration such as email, or Microsoft Teams HitHub integration bot.
Alternative e-mail based or webhook based notifications can be added via the utility script Send-Notifications.ps1.

Sample GitHub Actions Workflow Run

Sample GitHub Actions Workflow Run Test Results

Sample GitHub Actions Workflow Run Custom HTML Report

Sample GitHub Actions Workflow Run Custom HTML Report Published to GitHub Pages

Usage

Fork or Create your own Clone of this Repository
Update endpoints.json with your own endpoints
Setup your GitHub Actions Workflow and/or Azure DevOps Pipeline in your repository or Azure DevOps project.
*Optional:* Update the alertThresholds.json with your own defined values. This config file defines how many days of expiry left are marked as Low, Medium and High alerting severities.
Optional: If some of the endpoints in your list consists of network protected endpoints, not accessible to Microsoft hosted Azure DeVOps Pipelines or GitHub Actions Runners, please update the agent configuration as needed to use your self-hosted agents.
Optional: Update the Cron Schedule expression as required.
Optional: Update the Send-Notifications.ps1 script to directly send notifications from the pipeline. This may require you to also pass secret(s) to your pipeline in order to use the communication integrations.

Contributions & Issues

Contributions are welcomed, please raise a Pull Request with your proposed modifications if you wish to make any changes.
If you find an Issue and wish to report it, please use the Issues section.

License

MIT Licensed, please see license for details.

Practical PowerShell Scripting for DevOps - Part 4

MertSenel — Fri, 22 Oct 2021 13:38:34 +0000

Hi, I'm back with you with Part 4 of this series. On this part, I will show you how to check if a particular Git Reference belongs to a certain folder and also has a valid Semantic Versioning format. This type of script is a utility function that helps to fill gaps in particular automation scenario especially in deployment pipelines. Another common term used for these scripts is Glue Scripts. Glue in this context means filling a gap in your toolchain with a custom script and achieve your goals.

As DevOps engineers, we like to use complete and mature CI/CD tools. During my career I had used various CI/CD tools such as Jenkins, TeamCity, Azure DevOps, Octopus Deploy and Github Actions. Although all of these tools has rich repository of official and vendor provided tasks/actions almost all of them has common features to allow you run your own custom scripts. When pre-defined tasks comes short or you can't necessarily find something that does the trick out of the box, then you have go back to your biggest strength, writing your own custom glue script(s) to fill in the gaps.

Most of these CI/CD tools comes with a concept called predefined variables. These are system generated environment variables that are passed to your runners/workers that allows you to tap into certain platform or repository related metadata depending on the unique pipeline run. Today we will be looking at some sample GitHub Reference strings and find out if they are under a certain folder/path and also has valid Semantic Versioning format.

Semantic Versioning, or in short SemVer format, is a artifact versioning system that is widely accepted that allows engineers who write and maintain software in distinct versions. It is so widely used you've most likely came across this format in one shape or form.

Let's say you are releasing a product or a software service for the first time, your typical version number for this particular release can be 1.0.0 then let say your next release with some features add can be 1.1.0. Let say you've detected some defects and want to push some patches to second release, then you can name your new version as 1.1.1. Next release would be 1.2.0 and so on.

Going from 1.x.x to 2.x.x denotes a major version upgrade, of then denotes changes that are MAJOR and to point out that you are doing changes which contains good amount of change in the software. This is the simplest explanation of Semantic versioning and I can't possibly explain how Semantic Versioning works fully but I urge you to check out this page to fully understand it and google some more to learn about it. https://semver.org/

I'm going to give examples from GitHub and Azure DevOps but I'm almost certain that all CI/CD tool that works with Git repositories would have something similar.

Check out the pages for each product for predefined/environment variables available to you in their pipelines contexts.

Azure DevOps: https://docs.microsoft.com/en-us/azure/devops/pipelines/build/variables?view=azure-devops&tabs=yaml

GitHub Actions https://docs.github.com/en/actions/learn-github-actions/environment-variables

To simulate that we have one of these variable available to us in our script context, we will start of with a sample input array.

$refsInputArray = @("refs/heads/releases/1.0.0",
                    "refs/heads/releases/1.a.0",
                    "refs/heads/releases/1.2.0.5.6.9",
                    "refs/heads/release/1.1.1",
                    "refs/heads/releases/1.0.1",
                    "refs/heads/releases/2.4.3",
                    "refs/heads/main",
                    "refs/heads/develop")

As you can see, we don't get to choose the format we will receive our String Inputs, hence we need to know how to get certain parts of interest in a complex string like this and then perform our checks.

Without further due, lets get in to our problem statement and requirements in a real use case DevOps scenario.

Problem

Hey fellow DevOps Engineer! We have a pipeline for an API service we have, and we have a delivery pipeline that allows us to deploy our code to our serverless hosting platform from a particular branch of our choosing.

Although we give our developers flexibility to deploy to all environments from any feature branch they choose to deploy from, we would like to protect our Production environment. We have strict branch protection rules around releases/* branches and only wish to be able to deploy from these branches to our Production workers. We also use Semantic versioning to version our releases and cut our release branches with the version number of that particular release.

We've look into our CI/CD tool's out of the box and marketplace tasks, but couldn't find anything particular that can achieve what we are looking for, hence we need your help.

We would like you to write us a PowerShell script to perform a simple test to verify if the Git Branch we are running the pipeline from is actually adheres to our release versioning rules, this would save us from deploying the wrong branch to the Production environment.

1- We would like you to first check if the branch we are deploying from is under releases/ folder.

2- We also like to check if the branch name actually adheres to Semantic Versioning format.

What Success Looks like for this Challenge

Normally, each pipeline run would have singular input for a Git Reference hence the script would normally finish execution with a proper exit code.

Tip: exit 0 means successful execution and exit 1 means an exection with an unhandled exception or in other words a failure.

But because we are doing a mock challenge, in this example, I want you to start your script with the sample input array I've provided above and then simply return an appropriate message to the terminal stating if the input string fulfills the requirements or not.

Successful Run:

Some Pointers For Writing Your Own Solution

So let's breakdown the challenge a little bit. First of all we need to identify the folder of the branch.

In order to do this, we need the portion of the substring between the last two / characters. There may be multiple way of doing this so try to figure out how to achieve this first, once you have that portion then you can actually perform a string comparison to check if it's equal to releases

If your input passed the first requirement, then basically you need to capture the substring after the last / character. Once you have that then you can't simply do a string comparison as semantic version number can be multiple things, we want to check if it adheres to a pattern.

PowerShell being a scripting language has operators to make your life easy and allow you to achieve this in various ways, you can devise a PowerShell flavored solution. The most bulletproof method is to use something called regular expressions

At the bottom of the page I've provided above on semantic versioning they also provide regular expression to check if a certain string adheres to semantic versioning.

I'm using the second one in section https://semver.org/#is-there-a-suggested-regular-expression-regex-to-check-a-semver-string

^(0|[1-9]\d*)\.(0|[1-9]\d*)\.(0|[1-9]\d*)(?:-((?:0|[1-9]\d*|\d*[a-zA-Z-][0-9a-zA-Z-]*)(?:\.(?:0|[1-9]\d*|\d*[a-zA-Z-][0-9a-zA-Z-]*))*))?(?:\+([0-9a-zA-Z-]+(?:\.[0-9a-zA-Z-]+)*))?$

You can also script you own logic to check is this particular string you have adheres to the format by looking at it's components and if it has numerical values separated by dots in between. However, learning regular expression will help you in various string grouping and filtering scenarios in your cloud and DevOps career so I really urge you to learn and try to understand what they are and how they are used.

Event most seasoned engineers are struggling with regular expressions and so don't be discouraged if they appear to you as too complex at first. Even a small bit of knowledge goes a long way.

The Main Learnings From the Challenge

The main learnings from this challenge are:

1- PowerShell comparison operators

2- Regular Expressions

3- PowerShell String operator split to divide a certain string into substrings

4- PowerShell Arrays and how to target a particular Array Index

Links:

Matching operators

about_Regular_Expressions

about_Split

about_Arrays

Tasks You Need To Perform In Your Script

So to give another breakdown of what you need to do in your script,

1- Loop through the input array to check each input string individually

2- Figure out a way to isolate the substring between the last two / characters check if it's equal to releases

3- Figure out a way to isolate the substring after the last / character and check if it has a valid Semantic Versioning format (I recommend using a regular expression/regex pattern to perform this as it would also familiarise yourself with regex)

4- Print out relevant log messages to terminal that allows you to clearly see the input being evaluated and on failure, what requirement it fails and if satisfies both conditions a relevant success message. (Normally you would either fail or proceed with the pipeline runs according to the result but just printing a message is fine for this mock challenge)

Final Tips

From this point below you will see my sample solution. Depending on your level and learning style, either take a look before or after you gave it your shot.

My Sample Solution

Files for my Sample Solution

Part-4 Files

Test-SemVerWithRegex.ps1

$refsInputArray = @("refs/heads/releases/1.0.0",
                    "refs/heads/releases/1.a.0",
                    "refs/heads/releases/1.2.0.5.6.9",
                    "refs/heads/release/1.1.1",
                    "refs/heads/releases/1.0.1",
                    "refs/heads/releases/2.4.3",
                    "refs/heads/main",
                    "refs/heads/develop")

#https://semver.org/#is-there-a-suggested-regular-expression-regex-to-check-a-semver-string
$semVerRegex = '^(0|[1-9]\d*)\.(0|[1-9]\d*)\.(0|[1-9]\d*)(?:-((?:0|[1-9]\d*|\d*[a-zA-Z-][0-9a-zA-Z-]*)(?:\.(?:0|[1-9]\d*|\d*[a-zA-Z-][0-9a-zA-Z-]*))*))?(?:\+([0-9a-zA-Z-]+(?:\.[0-9a-zA-Z-]+)*))?$'

$refsInputArray | ForEach-Object {
    $refParts = ($PSItem -split '/')

    if ($refParts[-2] -eq 'releases') {
        if ("$($refParts[-1])" -match $semVerRegex) {
            Write-Host "Reference $($PSItem) is in releases folder and has a correct semver format"
        }
        else {
            Write-Error "Reference $($PSItem) is in releases folder but has an incorrect semver format"
        }
    }
    else {
        Write-Error "Reference $($PSItem) is not in releases folder"
    }
}

This is my solution and as I've stated during this part, there are multiple way of structuring your script and achieving the asked outputs. Only logic you are writing matters here, you can choose to have one complex if statement, you can use one complex conditional or opt-in for a switch statement for some more challenge, as long as you are fulfilling the challenge requirements you've accomplished the learning goals of this challenge.

Sample Runs

As our script is deterministic, it should always return the same result with the given input.

Conclusion

No matter, the purpose, the fundamental and primitive data types such as strings and arrays are things you will come across in various context no matter what environment you work in. Having a good understanding about these concepts are always beneficial and will allow you to increase your speed in devising solutions for complex problems.

Strings in particular plays a crucial role in DevOps scripting, Infrastructure as Code and pipeline automation as we usually parameterize certain templates and provide string based parameter inputs to those templates to achieve scalable automation solutions. Parametrizing templates and promoting along the software development lifecycle environments with one consistent template with different parameters will give you confidence in going to Production as you will be sure that you are deploying and testing the same set of resources in your lower environments.

This is a mock challenge to give you a medium to learn the skills to be able to work with string and arrays with Powershell, but as you continue along you will see they will come in handy in all type of different situations. I hope I was able to help you learn something new about PowerShell today, and I'll see you on the next one.

Practical PowerShell Scripting for DevOps - Part 3

MertSenel — Sun, 19 Sep 2021 04:06:49 +0000

Hi, I'm back with you with Part 3 of this series. On this part, I will show you how we can run loops in parallel for operations that doesn't have any racing conditions or dependencies on each other. This is a fairly new feature introduced with PowerShell 7.1+ and it's most useful to speed up your scripts. As this is a continuation of a series I highly recommend reading Part-1 and Part-2 first.

As I like to combine different concepts together, I will also combine the parallel loop approach with "Jobs" to show you how to make your scripts "Asynchronous". Combined, these concepts can be reused in various scenarios to help you write smarter and faster scripts.

As we are looking for performance improvements in terms of execution time, I'll also include a method you can use to measure and display execution times of your powershell scripts.

Problem

Hey fellow DevOps Engineer! The last script improvement was magnificent, no more false positive failures with the retry mechanism you've implemented. However we now have another problem! As we've introduced a retry mechanism into our HttpTest function, the script started to take a lot of time execute as each website is tested sequentially. The impact is felt more as our company grew further and now we have more endpoints we wish to perform these tests on.

1- We don't care in which order we got our test results as long as we have the results as fast as possible.

2- The worker infrastructure we are running these tests on are powerful enough to run 50 tests at a time.

3- We wish to re-use helper function "New-HttpTestResult" as is or with little modification as possible.

4- Can you please add another output at the end of the script to display the Total duration of the full script execution, this will help us confirm the performance improvements.

What Success Looks like for this Challenge

I've quickly added the Timer feature to our Part-2 Version of the script which doesn't have our new and improved parallel loop.

To fully appreciate the performance gains we will achieve from our improvement, I've also doubled our Test.json file to Test 18 endpoints.

As it can be seen from the screenshot, the total execution time is 113 seconds, it's quite a long time to wait, but this version of the script's loop is working in sequence and is synchronous by design, it has to wait for test at hand to go to the next one.

Here is the improved version:

First of all, same amount of endpoints only took 20 seconds to test with this version of the script. Another thing to notice is that, the order of results are not same as the order of our input, but instead it's in the order of test response return time.

As you can see, the tests which required more retries, hence took more time are displayed further down the list, whereas endpoints which returned a successfully result on first attempts are at the top.

Back to our original point, new version have improved the script's execution duration from 113 seconds to 20 seconds, it's an x5 times improvement. This will scale even further if the input list gets bigger.

Some Pointers For Writing Your Own Solution

This can be your starting point Part-2 Files of my sample solution (With Timer added)
Part-2 Files

Some Test Endpoints You Can Use

As this part I'll provide the starting point, you will already have the curated test endpoints json file readily available. It has multiple endpoints for simulating different scenarios. For more information please read part 1

The Main Learning From the Challenge

We are using new parallel feature of the ForEach-Object cmdlet and Thread Jobs which are another feature of the new PowerShell 7.

So I recommend reading and understanding these concepts from the documentation.

Finally, parallel foreach loops doesn't support running custom functions defined outside of the loop, as each iteration of the loop is a new runspace. Hence we need to define our helper function in each of the loop iteration. Currently, cmdlet doesn't supports this natively but there is a very easy workaround I've found on stackoverflow and sharing with you so you don't have to look for it.

The main idea is to, import your function a plain string, and then actually creating the function definition inside the loop iteration via using this string as source. Then you would be able to use your custom function(s).

I've used Stopwatch class to measure the script execution time. Stopwatch has been created via using the built-in class of .NET Core.
PowerShell can create objects from these classes via a syntax called Type Accelerator.

Links:

ForEach-Object
about_Thread_Jobs

How_to_Run_Custom_Function_in_Parallel_ForEach_Loop

StopWatch Class

about_Type_Accelerators

Tasks You Need To Perform In Your Script

1- Convert the foreach loop we had in Part-2 script to it's ForEach-Object -Parallel implementation.

2- Be able to reference and use your custom New-HttpTestResult function inside the parallel foreach-object loop.

3- Make sure you are displaying results as they come, rather than waiting for all tests to finish executing. You will need to implement Powershell Thread Jobs to achieve this behaviour.

4- Add a mechanism to measure and display the total execution time of the script

Final Tips

From this point below you will see my sample solution. Depending on your level and learning style, either take a look before or after you gave it your shot.

My Sample Solution

Folder Structure

Files for my Sample Solution

Part-3 Files

Tests.json

This is the list of endpoints I would like to test, stored in JSON format which can be produced and consumed with a variety of modern languages.
What we have here is an array of objects that has name and url keys for us to perform and label the tests.

As this file has been covered in last two parts I won't be providing more explanation.

Our Helper Function, create this file under path ./lib/New-HttpTestResult.ps1

As this file has been covered in last two parts I won't be providing more explanation.

Test-HttpEndpoints.ps1

Our Main script, and all the changes we've performed are on this script for Part-3 of the series.

I've introduced a Stopwatch object. Stopwatch has been created via using the built-in class of .NET Core.
StopWatch Class
PowerShell can create objects from these classes via a syntax called Type Accelerator.
Our foreach loop, now converted into a ForEach-Object Parallel version and we are running each operation as a Job.
Hence, now the result of our loop is not a collection of test results, but instead a collection of background jobs that have been started in parallel.
Now that we have the collection of Job created, we can wait for the results of these jobs. The results of these jobs, will be result of a HttpTestResult.

From there on, we print the test results as each job is finish running and gets completed.

[CmdletBinding()]
param (
    [Parameter(ValueFromPipeline = $true)]
    [string]
    $TestsFilePath =  '.\Tests.json'
)

#Lets Create a StopWatch Object.
$Stopwatch = [System.Diagnostics.Stopwatch]::new()

# Now Start the timer.
$Stopwatch.Start()

# Convert JSON Config Files String value to a PowerShell Object
$TestsObj = Get-Content -Path $TestsFilePath | ConvertFrom-Json

# Import the Tester Function
. ./lib/New-HttpTestResult.ps1

$funcDef = ${function:New-HttpTestResult}.ToString()

$jobs = $TestsObj | ForEach-Object -Parallel {

    ${function:New-HttpTestResult} = $using:funcDef

    New-HttpTestResult -TestArgs $_
} -AsJob -ThrottleLimit 50

$jobs | Receive-Job -Wait | Format-Table

# Now Stop the timer.
$Stopwatch.Stop()

$TestDuration  =  $Stopwatch.Elapsed.TotalSeconds

Write-Host "Total Script Execution Time: $($TestDuration) Seconds"

Sample Runs

Conclusion

The biggest benefit of Automation is that it has the ability to scale. However, while writing our scripts, we also need to consider the input it will take and the nature of operation we are performing.

Running loops in parallel, and using background jobs instead of waiting the execution of each operation is a way to speed up automation workflows. Imagine you are responsible for writing an automation to shutdown 10s of VMs on Friday evening. Would it really make sense to write a script that shuts them down, one at a time? What if that number was around 1000s, how would your automation scale?

Where possible, I tend to now leverage parallel loops to speed up execution and avoid unnecessary waiting in my scripts. Of course not each scenario can work with this type of loop, but deciding when you need it is also on you as the engineer. It's another tool in our toolkit, it's now up to us to decide where and how to use it.

Practical PowerShell Scripting for DevOps - Part 2

MertSenel — Sun, 23 May 2021 12:56:55 +0000

On this part, I will be extending the monitoring script we wrote on Part-1, with a polling while loop

Hi, I'm back with you with Part 2 of this series. On this part, I will ask you to extend the http ping monitoring script we wrote on Part-1. If you've completed Part-1 challenge then you extend your own script as well. In any case I will be providing my version of Part-1 solution as a starting point. Ok, lets jump on the our problem statement for this challenge.

Problem

Hey fellow DevOps Engineer! We loved the first script you wrote for us, now we can test the health of 10s of our endpoints with ease with just a configuration file. We now have a problem though, we can see that lot's of failures appears to be intermittent, as in if we simply retry couple of more times we will eventually get a successful result. We believe this behaviour is happening either due to a server warming up after a new deployment or some other delay throughout the network connectivity.

To filter out these FALSE positive failures, we would like the new version of this script to have a retry mechanism, so we can truly find out if a service is down. However, last time another engineer setup something similar, which we were running on a CI/CD pipeline, and for a unhealthy endpoint pipeline run would never end, costing our company a lot of money due to CI/CD agent hour costs.

So you need to make sure your solution meets the below requirements:

1- Retry the test for a maximum of N amount of times (configurable) this endpoint if the first attempt does not return a HTTP 200 response

2- Capture the attempt number of successfully response, if endpoint ever returns success code

3- Wait X amount of seconds in between retry attempts

4- Move on to the next endpoint if the maximum retry amount is reached

5- Prints results to terminal in a human readable format

What Success Looks like for this Challenge

The script I wrote on Part-1 were printing a result similar to:

Now we want our script to print a result similar to below:
Now on the first run of my script, I've configured the max retry parameter's value as 2, hence only with 2 retries not all of the flaky sites returned a successful result.

Increasing the max retry count to 10, we can see more endpoints now returning successful responses, on retry attempts. In this run for example FlakyWebsite-#4 endpoint return a success code on 9th attempt.

In both runs FaultyWebsite-#1 does not return a success code (as it's supposed to) and our script polls this endpoint for maximum amount of times.
Our solution works as expected. Great!

Some Pointers For Writing Your Own Solution

This can be your starting point Part-1 Files of my sample solution
Part-1 Files

Some Test Endpoints You Can Use

The Main Learning From the Challenge

Polling is a well known programming concept. You check state of a system, evaluate the result and make your next move. We can use loops to implement this retry logic in our script. However, in order to avoid the concept of "infinite loop" we need to make sure we are also implementing some limits and proper break conditions for our loop.

I recommended reading the Powershell documentation for while loop and break from their docs.
Links:

about_while

about_break

I'm using a while loop for these type of scenarios, but other loop types will work too. It will only change your implementation but same result should be achievable.

Tasks You Need To Perform In Your Script

1- Initialize Parameters/Variables for the Inputs you need. You need two new input, one for Maximum Amount of Retries and another one for Time to Wait in between retries.

2- Put testing part of your code in a loop, after your test evaluate the result with a conditional statement (hint: if/else block) and then either exit the while loop (hint: break)

of if a retry is needed, wait for the configured amount of time before making the next test attempt.

3- Initialize a counter variable and keep track of the no of attempt you are currently trying. If your test attempt is successful, you need to pass this attempt no back to the main script. Basically we want to if endpoint returned successful response after how many attempts. Example: 4/5 4 being the attempt number that we got a successful response and 5 being the maximum count of retries we would like to make.

Final Tips

From this point below you will see my sample solution. Depending on your level and learning style, either take a look before or after you gave it your shot.

My Sample Solution

Folder Structure

Files for my Sample Solution

Part-1 Files

Tests.json

[{
        "name": "FlakyWebsite-#1",
        "url": "https://httpbin.org/status/200,403,404,500"
    },
    {
        "name": "StableWebsite-#1",
        "url": "https://httpbin.org/status/200"
    },
    {
        "name": "FlakyWebsite-#2",
        "url": "https://httpbin.org/status/200,403,404,500"
    },
    {
        "name": "StableWebsite-#2",
        "url": "https://httpbin.org/status/200"
    },
    {
        "name": "StableWebsite-#3",
        "url": "https://httpbin.org/status/200"
    },
    {
        "name": "StableWebsite-#4",
        "url": "https://httpbin.org/status/200"
    },
    {
        "name": "FlakyWebsite-#3",
        "url": "https://httpbin.org/status/200,403,404,500"
    },
    {
        "name": "FlakyWebsite-#4",
        "url": "https://httpbin.org/status/200,403,404,500"
    },
    {
        "name": "FaultyWebsite-#1",
        "url": "https://httpbin.org/status/500"
    }
]

Our Helper Function, create this file under path ./lib/New-HttpTestResult.ps1

This script will perform an HTTP Ping Test with passed parameters and returns a Test Result Object

Notice the difference from the version in Part-1, this version has 2 new parameters and a while loop, to retry the test if

a non success code returns from the earlier responses. This is the same code you would write to poll anything, you would only change the code

that performs the test. The test can be a kubectl command that grabs a number of pods from a deployment replica set, and you would like to wait

for all containers are in ready state after a manual upscale in number of pods. Or you can perform a SQL connection or TCP port ping to validate

health in another type of non HTTP service. Possibilities are endless, but the main structure of the code will remain the same.

function New-HttpTestResult {
    param (
        [Parameter(ValueFromPipeline = $true)]
        [PSCustomObject]
        $TestArgs,
        # Maximum Retry Amount
        [Parameter()][int]$MaxRetryNo = 10,
        # Time to wait in between retry attempts
        [Parameter()][int]$WaitTimeInSeconds = 1
    )
    $ProgressPreference = 'SilentlyContinue'

    $Method = 'Get'

    $TestCounter = 0 

    # -lt: Lower Than 
    while ($TestCounter -lt $MaxRetryNo) {

        #Increment our counter by 1 before we make our first attempt
        $TestCounter++
        $duration = Measure-Command {
            $Response = Invoke-WebRequest -Uri $TestArgs.url -Method $Method -SkipHttpErrorCheck
        }

        # If we find the 200 code we stop polling
        if($Response.StatusCode.ToString() -eq '200'){
            break;
        }
        else {
            #Else we need to wait for configured amount of time
            Start-Sleep -Seconds $WaitTimeInSeconds
        }
    }

    $result = [PSCustomObject]@{
        name               = $TestArgs.name
        status_code        = $Response.StatusCode.ToString()
        status_description = $Response.StatusDescription
        attempt_no         = "$($TestCounter)/$($MaxRetryNo)"
        responsetime_ms    = $duration.Milliseconds
        timestamp          = (get-date).ToString('O')
    }



    return $result
}

Test-HttpEndpoints.ps1

Our Main script, we will invoke this script to run the tests and produce and returns a test results array

[CmdletBinding()]
param (
    [Parameter(ValueFromPipeline = $true)]
    [string]
    $TestsFilePath =  '.\Tests.json'
)

# Convert JSON Config Files String value to a PowerShell Object
$TestsObj = Get-Content -Path $TestsFilePath | ConvertFrom-Json

# Import the Tester Function
. ./lib/New-HttpTestResult.ps1

# Loop through Test Objects and get the results as a collection
$TestResults = foreach ($Test in $TestsObj) { 
    New-HttpTestResult -TestArgs $Test 
}

$TestResults | Format-Table -AutoSize

Sample Runs

Conclusion

In automation scenarios, you will want to implement defensive scripts, that are not a one shot series of instructions but it has some more bells and whistles to make it more prone to environmental changes in your infrastructure. Having a retry mechanism to wait for a system's state to reach a desired state is a very common need in DevOps automation tasks.

I hope in this part I was able to help you learn something new. If you are giving an attempt to the challenges, good on you. If you are just reading my solutions, do not feel guilty. I have and still reading a lot of other engineer's code on GitHub and their blogs. If you are currently not at a level to give the challenge a try yourself, download my sample, run it and try to understand how it works. Then if you can try to change something in it and see if you can change the behaviour of the script.

I currently don't know what the next part will be about, I'm thinking about running our tests in a CI/CD pipeline or writing another script to create the json file with our endpoints listed automatically, via scanning an Azure subscription. Both of them can come in any order as they are not dependent on each other. I'll see you on the next one.

Practical PowerShell Scripting for DevOps - Part 1

MertSenel — Sat, 27 Mar 2021 08:12:51 +0000

Hello, friend. Welcome to Practical PowerShell Scripting for DevOps. In this series, I will attempt to teach to you how to write PowerShell scripts and solve some real-life DevOps challenges by writing scripts in PowerShell language.

Scripting or Programming in general is a must have skill for anyone who want to seek a Cloud or DevOps career today. This series is geared towards giving some guidance on how to create your own PowerShell Scripts yourself.

Who Is This Series For?

DevOps engineers are tasked to operate in cloud environments with great scale of resources. Often times, scripting is an easy way to automate tasks that can be programmatically achieved to save time and produce consistent results.

If you are trying to learn Powershell for getting into a Cloud / DevOps career path and struggling to find some exercise challenges to solve.

This is not a best practices or formal PowerShell education resource, I merely try to provide some made up DevOps/Cloud scenarios that can be used to create some Powershell scripts in an attempt to practice and learn scripting skills.

I suggest, using the Powershell reference documentation as much as possible. Spend time on terminal and play around with different cmdlets.
Your ultimate goal should be trying to learn a style for designing and creating a automation solution for any scenario.

The tools you will learn and pickup along the way while solving these scenarios will accumulate and will widen your horizon in terms of what you can do with automation and scripting.

Problem

Imagine you are DevOps engineer for a project with multiple public web endpoints exposed to Internet. Your hosting provider is having an intermittent network outage on their load balancing infrastructure. You need to write a powershell script:

1- That accepts a list of urls to perform Http Ping Test (Http Get Request) from a configuration file
2- Performs a Http Get request test for given list of endpoints and return test results containing

Status Code Returned Http Response
Status Description For the Http Response
Response time of the request
Timestamp for the test result

as a collection

3- Prints results to terminal in a human readable format

Some Pointers For Writing Your Own Solution

Some Test Endpoints You Can Use

Although you can use any public endpoint, I'm using https://httpbin.org status endpoint to randomly return me an HTTP status code for simulating a real testing scenario. When we hit a test url with multiple status codes aka Flaky Sites, we will receive different results.

if you use "https://httpbin.org/status/200,403,404,500" url you will randomly get a response code back, 200 being successful and other response codes simulating different types of failures.

you can use "https://httpbin.org/status/200" to only receive successful test results. These endpoints can be your starting point to use as a playground to curate your script.

The Meat Of The Solution

Usually every script has a core functionality cmdlet you want to use, development usually starts with the results received from this core cmdlet.

You can perform HTTP request in different ways in PowerShell, but most common ones are using the cmdlets:

Invoke-RestMethod and Invoke-WebRequest

I suggest starting reading documentation for these cmdlets first, try to use them to perform some tests manually in terminal first, capturing the result and then curate a plan from there.

Input And Output For Your Solution

I've placed a requirement to prepare and consume the list of endpoints to tests from a file. You can use any format you like, row separated text file, a CSV file all are good.

I've used and recommend JSON as it is the most richest and universal notation that can be used amongst a lot of programming languages.

This skill once learnt will come in handy for you in various scenarios where you will need to handle a set of data, you can't just store and handle as a command-line argument.

Tasks You Need To Perform In Your Script

1- Get your Input
Try to figure out individual tasks you need perform first to get to a state you can actually do the work.

In our case, you want to end up with your list of endpoint. To achieve this, we need to access our config file that holds our test endpoints.

Maybe have a parameter for accepting the file's path and then consume the content of this file.

2- Have a function to perform your Test
It is good practice to create a function for repetitive code invocations, in our case we know that we will run test and produce a test result for multiple endpoints, hence it's a good idea to separate this part of our code to a function.

3- At this point you have everything you need

Loop through your test endpoints, invoke your test function for each and capture the results.

4- Print the result

Display the results, with way of your choosing.

Final Tips

From this point below you will see my sample solution. Depending on your level and learning style, either take a look before or after you gave it your shot.

My Sample Solution

Folder Structure

Files

GitHub

Repository Url

Tests.json

[{
        "name": "FlakyWebsite-#1",
        "url": "https://httpbin.org/status/200,403,404,500"
    },
    {
        "name": "StableWebsite-#1",
        "url": "https://httpbin.org/status/200"
    },
    {
        "name": "FlakyWebsite-#2",
        "url": "https://httpbin.org/status/200,403,404,500"
    },
    {
        "name": "StableWebsite-#2",
        "url": "https://httpbin.org/status/200"
    },
    {
        "name": "StableWebsite-#3",
        "url": "https://httpbin.org/status/200"
    },
    {
        "name": "StableWebsite-#4",
        "url": "https://httpbin.org/status/200"
    },
    {
        "name": "FlakyWebsite-#3",
        "url": "https://httpbin.org/status/200,403,404,500"
    },
    {
        "name": "FlakyWebsite-#4",
        "url": "https://httpbin.org/status/200,403,404,500"
    },
    {
        "name": "FaultyWebsite-#1",
        "url": "https://httpbin.org/status/500"
    }
]

Our Helper Function, create this file under path ./lib/New-HttpTestResult.ps1

This script will perform an HTTP Ping Test with passed parameters and returns a Test Result Object

function New-HttpTestResult {
    param (
        [Parameter(ValueFromPipeline=$true)]
        [PSCustomObject]
        $TestArgs
    )
    $ProgressPreference = 'SilentlyContinue'

    $Method = 'Get'

    $duration = Measure-Command {
        $Response = Invoke-WebRequest -Uri $TestArgs.url -Method $Method -SkipHttpErrorCheck
    }

    $result = [PSCustomObject]@{
        name = $TestArgs.name
        status_code = $Response.StatusCode.ToString()
        status_description = $Response.StatusDescription
        responsetime_ms = $duration.Milliseconds
        timestamp = (get-date).ToString('O')
    }

    return $result
}

Test-HttpEndpoints.ps1

Our Main script, we will invoke this script to run the tests and produce and returns a test results array

[CmdletBinding()]
param (
    [Parameter(ValueFromPipeline = $true)]
    [string]
    $TestsFilePath =  '.\Tests.json'
)

# Convert JSON Config Files String value to a PowerShell Object
$TestsObj = Get-Content -Path $TestsFilePath | ConvertFrom-Json

# Import the Tester Function
. ./lib/New-HttpTestResult.ps1

# Loop through Test Objects and get the results as a collection
$TestResults = foreach ($Test in $TestsObj) { 
    New-HttpTestResult -TestArgs $Test 
}

$TestResults | Format-Table -AutoSize

Sample Runs

On each run, there is chance to get a different result back for flaky test endpoints.

Conclusion

I hope this sample challenge helped you out learning powershell a bit and give it more context on how you can craft custom tools, in a short time frame.

You may share your solutions, via the comments if you wish me to review them or have any questions.

In Part 2 of this series, I am planning to go into Azure Cloud and I will share another challenge with usage of Az PowerShell module.
Hope to see you there.

Agile Cloud DevOps Engineer

MertSenel — Sun, 03 Jan 2021 11:18:49 +0000

Designs like a Cloud Architect, Ships like a Developer, Operates like a Site Reliability Engineer My transformation in 2020, and what Agile/Embedded DevOps Engineer Model is all about

2020 is finally over and it has been a hell of ride for me and I guess for everybody around the globe. Although, it was not the best year in my life personally as I have suffered a great length of social isolation due to COVID and some unfortunate circumstances, I still manage to hold myself together, be productive and kept growing. I may not be at my happiest state, but I certainly didn't made this an excuse to stop myself from working extra hard to pursue my goals and dreams, while keeping up with my professional duties. I feel more resilient and confident as a result of all of it. Besides all the negative, career wise this year also felt like the most accomplished year for me.

I will try to keep this article focused on my transformation in 2020, and try to explain you what Agile/Embedded DevOps Engineer model/role is.

2020 Highlights

First, a short list of my major achievements:

New role as an Agile/Embedded DevOps Engineer
A great deal of improvement my technical skills in PowerShell, Azure Cloud, Azure DevOps, Docker, Infrastructure as Code, Automation, Monitoring, CI/CD and Automation.
Opened my first personal blog/portfolio website.
My first open source contribution to Azure quickstart repository.
Published 6 posts and 6 DevOps projects in my website.
My submission won the Azure Festive Tech Calendar Automation Deployment Hackathon

https://www.wesleyhaakman.org/festive-tech-hackathon/

I had connected and engaged with a lot of Cloud/DevOps/SRE professionals and content creators of all kind around the world via social media, learnt from what they are sharing and try to share my knowledge wherever I can to pay forward what I have been receiving online.

So What is Agile/Embedded DevOps Engineer

First of all, the exact terminology that we actually use in my company is "Embedded DevOps Engineer", Agile is something I've come up with so it resonates with a larger audience. If you have never heard this concept, don't worry I had not too until December 2019.

It all happened during my initial meeting with my current company's Vice President of Operations. He asked me, what do you think your role will be here like? I told him that I would oversee all DevOps related aspects of the project and pretty much drive the project, take it to the finish line.

As an operations engineer this is usually what is expected of you, being the owner of a particular project or a solution and delivering it end to end. Having "Root” level access to all, being keeper of secrets and systems.

His answer was short, he simply denied this approach and started to explain to me they are going to be implementing a new model called "Embedded DevOps” where an "Embedded DevOps” engineer(s) is a directly assigned resource of a software delivery team. Moreover, the DevOps mentality will be expected and adopted by all members of the engineering teams. So it will be a shared duty amongst everybody to operate the services.

The embedded engineers will also be engaged in a central "DevOps Practice” team that they would allocate a certain amount of time per week to work on common goals and practices that should be promoted to the rest of the engineering teams.

Then he asked me, what do you think about it? I said great! Let’s do it. Fast forward 12 months and it was surely one of the best work experiences I had so far.

Making the best use of your DevOps Talent

Having a group of talented DevOps engineers in your company is great, but they are no good to anyone if they are isolated from the application development processes. If they are kept aside in yet another silo labeled as “DevOps”. Instead having them embedded in the software development teams, would make their knowledge, skills and their unique perspective available to the software project, from day one.

Increasing the engagement also allows for software and QA engineers to learn DevOps concepts. Likewise, as a DevOps engineer I feel like I have learnt a lot in terms of software development too.

This is possible because there is almost no resistance in terms of accepting change. This is very important, that is why as DevOps engineers we shouldn't have to drive the change and DevOps adoption. Instead it should be everyone’s desire inside an organization to implement and practice DevOps while developing, delivering and supporting their products. Hence, a DevOps engineer's duty is more on enabling and improving these practices. Providing solutions to problems, where engineering teams need assistance.

What about company wide DevOps Challenges

This is where DevOps practice comes into play. Having embedded DevOps engineers are great, but this fills their day with development and operations tasks for their assigned product(s).

All DevOps engineers embedded or within the central DevOps team, all of them join once a week for a catch-up type of meeting where the structure is fluid. It may include, planning, announcements, discussion points and so on. Then, DevOps engineers in groups of four or five, split to work on different larger scope projects.

These projects are problems that are concerning the whole company. These are problems where a larger collaboration is needed either for designing or implementing a solution to a particular problem.

Key Requirements of the Agile/Embedded DevOps Model

Below are my key highlights of the necessary practices that needs be in place for a DevOps practice that works in this fashion:

DevOps from the beginning, designing 12 factor apps means our projects are suited for faster change delivery. We are able to deliver a change in a matter of hours if needed.
Security starts from code; with strong collaboration between security, software and DevOps engineers, source code, infrastructure and our application architecture is secured and hardened according to industry best practice starting from the development phase.
Proactive and pre-planned work; this model allows for better allocation of DevOps engineer’s time in terms of task planning. They are clear of their responsibilities and priorities. This allows for better planning for their managers and themselves. This is a much better allocation compared to reactive and random project assignments that leaves DevOps engineers overwhelmed and alienated.
Collaboration and transparency is encouraged across the company. Inside the company, information is free to flow, so everyone can learn. Repositories, Wikis, document sites are there for us to work better and find solutions faster.
Sharing meetings, demos and similar workshops are regularly held.
On-demand training resources are easily available and bootcamp type more structured training is offered on a regular schedule.
Using Infrastructure as Code and end to end Automated Continuous Delivery.

Design Like an Architect - Your Personal Cloud SME

One my of duties is, acting as an immediately reachable Cloud SME(Subject Matter Expert), to my team and managers. Being a cloud enthusiast, this is one of the areas I excel at, and I have great pleasure when I have to work with this nature of a task or a story.

This year, my role allowed me to curate and extend cloud infrastructure for both brownfield and greenfield services either we host as a part of our customer facing products, or a line of business service we are hosting internally, that we use for business intelligence or software development operations.

Taking part in these major decisions is very important as the decisions made at this stage can make or break your solution's:

Usability
Reliability
Scalability
Availability
Testability
Supportability
Security
Liability

Which are common features for almost all enterprise software we use day to day. No matter how good a software product is, it won't be successful without having the features listed above. If you have frequent outages, or you can't figure out your products scaling bottleneck which ends up in poor performance, it probably won't attract a lot of market value. These are now expected as a default baseline by software customers and DevOps really shines assisting in delivery of these features.

Ship Like a Developer - Scrum for Ops

One of the main differences in this working style is that, my work is also planned in 2 week scrum sprints. This forced me into breaking down my stories and tasks, have a plan and a list of deliverables, acceptance criteria in a well documented manner that works for us. Effort estimations were difficult at first but, the more you do it the easier it gets.

My work is reviewed and merged via Pull Requests and rest of the team is familiarized and kept up to date of the process during Scrum meetings.Working closely with the Application Developers, UI/UX and Product Management allowed us to have a much more efficient SDLC processes overall that boosted up everybody's effectiveness.

It's not my character to contain myself inside a box, so I usually overreached and gone beyond my duties in understanding business logic of the services I worked on, which in return presented itself as enhanced value to the team and the products we are shipping.

Another significant benefit of working inside a development team is, knowledge sharing. I have certainly improved my programming skills, Database/Query language skills, CI/CD skills for different technology and infrastructure stacks and overall my comfort level while bouncing ideas of other Developers. I'm also sure that on the other hand my teammates after working with me now have a better understanding in cloud infrastructure, networking and overall software operations.

Operate Like a SRE (Site Reliability Engineer) - Modern SysAdmin in NoOps Era

Coding and releasing infrastructure is great and fun, but the story doesn't usually ends there. A developer too can most likely put together some infrastructure as code or automation to bring up infrastructure for their projects, but performing as a SRE requires a unique set of skills and mindset which cannot be overlooked.

While developing an application, in conjunction we are also developing code/config that provides integration with monitoring and observability tooling for alerts and visibility, configuration management, secret management, release management and more.

Deployment patterns like blue-green deployments for stateless services, schema updates to your database layers these all require some unique expertise to be planned, automated and performed.

I use to be a system administrator that does the same for infrastructure of our own, but as more and more compute is being passed on the cloud service provider's "SysAdmins" we get to operate our "Services" instead of "Infrastructure".

So, we won't be concerning ourselves, for hosting a highly available web server or a database server, PaaS offerings already gives us much of these, but instead the overall harmony of our micro services, their integration, the SLA's and SKUs we require to pay in order to fulfil our solution's requirement. This does not mean we can't, if need rises we can but we simply choose not to whenever possible as it all adds up as toils.

We also don't concern ourselves by hosting our CI/CD tooling, source control or any other collaboration tool and opt-in for cloud native SaaS and PaaS offerings wherever possible. Anything that adds an admin effort is actually more costly to a business. Your customers don't care about how well you host your Jenkins clusters or Octopus Deploy server, all the value is in the delivered software so the focus should be on those tasks, not things where there is someone else doing it as a commercial product.

All of this is also coherent with the designing element of this role. You can think about the whole picture, including the operations duties while making the architecture choices for your application/infrastructure.

Knowing a service by it's nature is very important in it's uptime, health and reliability. For a messaging infrastructure like a Kafka server or a object storage service like Azure Storage Accounts, you may be only needing do a metrics based monitoring but what about an API that is sitting behind a WAF and requires OAuth authentication, you cannot simply ping your API, you need to know your service in order to monitor it's health state and performance. This is were SRE skills comes into play that incorporates all of the supporting services, side by side with the actual application/service.

Conclusion

Overall, I can say that this year was another year of great deal of learning, accomplishment, recognition and action taking. I'm very proud of myself to go through this transformation and came out fine on the other side. I may be wronged in the future but I believe, our limits are defined by the mental ceilings we put for ourselves. I always actively try to set a higher ceiling and standards for myself. The only way to reach those ceilings is believing in myself that I will get there and put in the effort necessary, no excuses and by all means. I admit that I'm my own biggest fan and I suggest you should be yours too. If you don't believe in yourself, how can you expect others to do so.

Microservices Lab on Azure Kubernetes Service

MertSenel — Sun, 04 Oct 2020 08:56:28 +0000

Microservices on cloud-based Kubernetes - Azure

Introduction

Inspired by the Original Project: https://github.com/didier-durand/microservices-on-cloud-kubernetes

This is an implementation of the same micro-services pattern deployment lab implemented with Azure Kubernetes Services.

(see Credits Section for more information).

The purpose of this repository is to provide the fully automated setup of a nice-looking (see screenshots)
showcase / testbed for a cloud-native (precisely defined application
by Microsoft) on a cloud-hosted Kubernetes cluster (here GKE by Google Cloud) based on an interesting service mesh.
So, additionally, the setup will install tooling (coming from the CNCF Cloud Trail Map for many of them) to make the application and its service mesh observable and manageable.

Another goal of this repository is to help people exploring the cloud-native architecture: when you fork it, you rapidly get a working cluster with a somewhat
"real-life" application and decent tooling to experiment with, without the need for a long trial-and-error process starting with infrastructure to set it
up from scratch. It makes it much faster to grasp the philosophy of the distributed architecture proposed by Kubernetes.

My Purpose for the AKS / Cloud Kubernetes Lab

My purpose was to create a Cloud Native Kubernetes Lab, for Operations, DevOps and Site Reliability Engineering Perspective.

One of the main problem for training complex infrastructure scenarios is that, it's not always possible to have an application workload to work with that can simulate real life scenarios, or close to a production workload in terms of complexity.

My purpose with this repository is to have a baseline for quick and fast start with a function Azure Kubernetes Lab.

Then use the infrastructure to work on more advanced DevOps concepts.

Some improvements I can think of right now:

1- Learn and Practice with the Observability and Monitoring Tools

2- Practice IaC and Pipeline Concepts, (GitHub Action provided here is not a good reference for proper CI/CD, they are quick and easy methods to keep everything contained to a GitHub Repository)
3- Practice Kubernetes Cluster Administration Operations, Upgrade Cluster Version, Scale-up, Scale-Out

4- Setup Advanced Networking for Ingress. Currently project used unprotected HTTP protocol, having the traffic secured via SSL can be an improvement.

5- Practice, more advanced load testing, stress testing and reliability testing (chaos monkey).

6- Learn how to co-host multiple team's operations on a single AKS cluster (developing different services) using AKS Dev Spaces and Kubernetes namespaces in general.
7- Build the application services from source, and customize them to integrate with Azure Application Insights for APM and Telemetry Logging.

Access to deployed tools & dashboards

Available dashboards:_

(click on pictures to enlarge them - also use the hyperlinks provided with each dashboard description to have a good overview of the
features of each tool from its official documentation

Standard K8s UI: our workflow deploys first this standard Kubernetes dashboard as a tool that should anyway be included in any installation. It gives a good overview of the deployed cluster with static (configuration) and dynamic (metrics) information about the active objects. When kubectl proxy is active, the dashboard is available at this url. The security check at login is most easily satisfied by selection the login option of config file (see Prereqs to obtain it in Setup section).

Azure Portal & Azure Container Insights Solution As per Azure Documentation Kubernetes Dashboard is soon to be deprecated and recommended solution is to use Azure Container Insights Solution which brings, the management plane visibility to cloud providers operations tools. This gives a better controls(in terms of access and security) and operations capability to the cluster operators. My implementation, deploy Azure Container Insights Monitoring Solution with the ARM template. ARM templates, is considered Infrasructure as Code and includes can include it's own workflow logic. Hence, the template deployment flow looks like below:

1- Deploys a Log Analytics Workspace

2- Deploy the Azure Kubernetes Cluster, with Monitoring Add-On enabled.

3- Deploys the Container Insights Management Solution to Log Analytics Workspace, created in step1.

Polaris dashboard: Polaris is an interesting tool, even in its OSS version (the paid-for version provides more checks) used here. After installation, it scans the definitions of various kinds of objects and applies sanity checking rules to validate their proper configuration. For example, in the case of Online Boutique, it will warn that containers have no resource constraints (cpu, memory, etc.) imposed on them or that their security credentials are too wide compared to what they do. The hyperlinks provided on the unsatisfactory checks document the reason(s) of the alert as well as the possible remedies to apply. So, a quite useful tool to incrementally increase the quality of the configuration of a given cluster: new versions of yaml object manifests can be gradually deployed to minimize the issue notifications.

Kiali Dashboard: Kiali claims itself the "Service mesh management for Istio". it allows an interactive discovery of the defined relations between the services. Then, it provides detailed insights and metrics about the health of those services and the requests between them. The live traffic animation in the UI is extremely useful: it allows to spot very quickly where the activity happens to focus on those hot spots during root cause analysis for an issue. You can also go back in time with the replay feature to see the traffic and the interactions that happened in the past to understand why and how you reached current situation. This dashboard is accessed via istioctl dashboard kiali that will open the corresponding UI into your web browser.

Grafana dashboard: Grafana which allows you to query, visualize, alert on metrics and logs, no matter where they are stored provides very nice charts about the activity of the cluster as the whole (usual metrics about resource consumption: cpu, memory, etc. for nodes. But, more specifically in this context it provides interesting additional dashboards specific to the Istio service mesh related to the traffic between the pods. Those dashboards are accessed via istioctl dashboard grafana that will open the corresponding UI into your web browser.

Jaeger dashboard: the Open Boutique is instrumented via OpenCensus, now merged into OpenTelemetry, component of the CNCF Cloud Trail Map. Jaeger - in CNCF Cloud Trail Map - is the tracing backend implemented here. It centralizes and ingests the distributed traces produced by the various microservices. So, the Jaeger dashboard will allow the detailed examination of those aggregated distributed traces also known as "spans". This dashboard is accessed via istioctl dashboard jaeger that will open the corresponding UI into your web browser.

Prometheus dashboard: Prometheus (also member of CNCF Cloud Trail Map) is the cornerstone component for metrics collection. The collected data is used by Grafana, Kiali & Jaeger for their specific purposes. The Prometheus dashboard can be used as the "source of truth": for example, it can be used to verify if some metrics claimed as missing by a downstream component using this value is really collected or not and to compare the graph produced by Prometheus itself to the graph produced downstream user to spot potential discrepancies. This dashboard is accessed via istioctl dashboard prometheus that will open the corresponding UI into your web browser.

Envoy dashboard: an improved version of Envoy Proxy, also part of the CNCF Cloud Trail Map, is the sidecar container used by Istio. Envoy provides a (somewhat rough) dashboard to go into the nitty-gritty details of a given pod: it is usually used for low-level introspection into the traffic between pods in unexpected situations. It is more a debugging tool at very low-level: most cluster administrators shouldn't need to use it. This dashboard is accessed via istioctl dashboard envoy podname[.namespace]: it will open the corresponding UI for the chosen pod sidecar into your web browser.

Workflow steps

The workflow has following steps:

checkout of project artefacts
Install Istioctl to the agent
Login to Azure Context and Install Tools Az Cli and Az Pwsh(subsequent steps in shell script)
Deploy Infrastructure, which includes the Azure side Monitoring Infra and Integration
import cluster config & credentials for kubectl & istioctl
deploy Polaris dashboard and check its proper deployment
deploy Istio and check its proper deployment
deploy Istio-based addons and check their proper deployment
label application namespace to ensure automatic sidecar injection (see Istio architecture uses an improved version of Envoy for sidecars) in microservice pods
~~deploy proper application manifests for Istio~~ Not need for Azure
deploy Online Boutique and check its proper deployment
run kubectl get all to display all deployed assets and IP address information

Application can now be accessed as described below

AKS Lab Demo User Snippets

You can start using your lab via instructions below.

Setup Project Resource Names

#Curate Variables
$ResourceGroupName = "mert-lab-aks-aue01"  #Update with your RG Name
$AksClusterName = "mertlabaue01-aks" #Update with your AKS Lab Cluster Name

Get your Kubectl Credentials

Import-AzAksCredential -ResourceGroupName $ResourceGroupName -Name $AksClusterName -Force

Get All Objects in Kubernetes Cluster

kubectl get all

Find the IP Address of the Online Boutique FrontEnd

$FrontEndService = kubectl get service/frontend-external -o=json | ConvertFrom-Json -Depth 100
$FrontEndServiceUri = "http://{0}" -f $FrontEndService.status.loadBalancer.ingress.ip

Write-Output "Online Boutique FrontEnd Uri: $FrontEndServiceUri"

Online Boutique FrontEnd Uri: http://52.147.9.72

Get Logs from Load Generator

kubectl logs -l app=loadgenerator -c main
 GET /product/66VCHSJNUP                                          600     0(0.00%)      77      34    1048  |      41    0.10    0.00
 GET /product/6E92ZMYYFZ                                          563     0(0.00%)      77      34    1763  |      41    0.00    0.00
 GET /product/9SIQT8TOJO                                          593     0(0.00%)      73      34    1013  |      41    0.30    0.00
 GET /product/L9ECAV7KIM                                          631     0(0.00%)      82      34    1349  |      42    0.20    0.00
 GET /product/LS4PSXUNUM                                          608     0(0.00%)      83      34     896  |      42    0.20    0.00
 GET /product/OLJCESPC7Z                                          623     0(0.00%)      69      34    1079  |      41    0.10    0.00
 POST /setCurrency                                                808     0(0.00%)      82      44    1089  |      51    0.20    0.00
--------------------------------------------------------------------------------------------------------------------------------------------
 Aggregated                                                      9517     0(0.00%)                                       1.80    0.00

Credits

Inspired by the Original Project: https://github.com/didier-durand/microservices-on-cloud-kubernetes

This is an implementation of the same micro-services pattern deployment lab implemented with Azure Kubernetes Services.

To read his full README and instructions click here. I will be moving relevant parts to this repo and update with new instructions, specific to our implementation.

License

Below is Didier's License notice, and I'm carrying over the same LICENSE, so you are also free to re-use the resources in the repository.

This application, licensed under Apache terms (same terms for all components used in this worklfow - So, allowing free reuse) is the "Online Boutique"
(formerly known as Hipster Shop - developed by a Google team but not an official product of them). It is composed of 10 polyglot microservices behind a nice-looking web frontend calling them to serve client requests.
A load-generator - part of the package - will generate traffic while the application is running to make use of tools (Prometheus, OpenTelemetry,
etc.) more attractive.

Application features & service mesh

application service mesh

This demo application contains an interesting service mesh to give some substance to demos and tests: its schema is given above. This mesh
which is thoroughly used by a traffic generator - also part of the demo package - which generates constant solid traffic to make the implementation
of monitoring tools.

Interesting points of Online Boutique:

Multi-language: the microservices corresponding to the various application features were on purpose written on purpose by the authors in
numerous languages (Go, NodeJS, Java, C#, Python) to demonstrate a key strength of container-based applications: many microservices collaborate in
a "polyglot" environment where each team (or individual) can program in its language of choice while ad hoc frameworks for each language make sure
that all Kubernetes standards (probes, etc.) and architecture can be easily respected with minimal effort to obtain a coherent and compliant global
system, manageable by the standard palette of Kubernetes tools. This polyglot aspect is reinforced by the mixed use of http and gRpc, which are both
understood by the monitoring tools.
Service Mesh: the application graph shows relationships between the various services and the front-end. Indeed, the application
is made of13 pods. This high-level of granularity is the accepted Kubernetes pattern for application architecture: it brings numerous advantages like continuous
delivery, exhaustive unit testing, higher resilience, optimal scalability, etc. But, it also requires the use of a thorough set of tools to
maximize the observability of the system. If "divide and conquer" is the motto of cloud-native architectures, the motto of their operations is probably
"observe to sustain": when working with Kubernetes application, one feels very quickly the need for (very) solid tools monitoring automatically
the myriad of objects (services, pods, ingress, volumes, etc.) composing the system.
GCP Tooling: the application is instrumented for Stackdriver (profiling, logging, debugging). So,
the source code of this application provides the right guidance to see how to code in order to obtain the right leverage on tools directly available
from the GCP service portfolio.

Setup for forks

To start with, you need an active Azure account that you can use the providers required for this project, like Kubernetes and Azure Monitor.

Then, fork our repository and define the required Github Secrets in your forked
repository:

All you need for this project is to setup one secret called "AZURE_CREDENTIALS".
Open the run the powershell script 'New-AzAksLabPrerequisites.ps1' you can find here

This script will create a Service principal, save the credential json file to your local as "AZURE_CREDENTIALS.json" and also create a new environment variable called "AZURE_CREDENTIALS" in your local system.

This local environment variable part is optional, but it makes it easier for you to also run the scripts locally, without using github actions.

Once you have the json file generated, copy and paste the contents of that file as your "AZURE_CREDENTIALS" Github Repository Secret.

To easily launch the workflow, you can launch it with the manual dispatch feature of Github that you can see as a launch button in the Action tab of your project for
the "Deploy AKS Lab Infra and Services" workflow. Similarly, you can stop it via similar button in "Clean-Up AKS Lab Resources" workflow.

If you already have deployed the Infrastructure but something went wrong during services installation you can run "Deploy AKS Lab Services Only" to skip Infrastructure deployment and directly go to service deployment.

When the deployment workflow completes successfully, you should be able to access the Online Boutique from anywhere on the Internet at the pubic IP
address displayed in the final lines of step "Deploy AKS Lab Infra and Services"

To get access to the cluster via kubectl see above

Finally, you should install kubectl and
install istioctl if not present on your laptop yet.

To install these on a Windows laptop, I've left a script 'Install-ToolsOnWindows.ps1' with snippets inside to show you how to install istioctl and kubectl on a Windows machine.
They are both stand-alone executables so there are many ways to install them.

Github repository for the project: https://github.com/MertSenel/microservices-on-cloud-kubernetes-azure

Lessons Learnt Along The Way

MertSenel — Sun, 27 Sep 2020 11:48:03 +0000

It has been 11 years past since my first full-time position as a "Junior Networks & Systems Administrator". As of writing this I'm 31 years old, hence my corporate IT career journey started when I was 20 years old. A bit younger than today's university graduates. Funny how time has passed and today I'm in a position where I assist and mentor juniors both formally and informally in a workplace environment.

Couple of months ago one of our managers came in and said they want to hire some "DevOps" interns and are planning to start a mentoring programme, where we pretty much come up with a curriculum of concepts and tools, we desire these interns to learn eventually. As a part of this programme, mentors who are already established DevOps engineers would become mentor's and provide assistance in those particular subjects in the curriculum.

Needless to say I've raised my hand to volunteer to be one of the mentor's and already put in some work to come up with learning topics, materials, a schedule and flow for the topic and related activities as such. However, yesterday I was thinking, besides from all the discipline specific knowledge, what can I actually teach to a junior that would benefit them the greatest. What advice, what insight, what gem I could give them, that would be of guidance in their journey in the next 10 years.

So it is yet again the million dollar question, if you can talk to your 20 year old self right now, what would you tell him or her about the upcoming 10 years.

The things I will share today are insights I have accumulated over the years and proven to be true again and again. However, I have this point of view about giving advice and knowledge sharing in general. If you go to anyone and just pour your wisdom on them, it never creates the effect you wish it does. According to my past experience, all advice I've received and actually applied are the answers I personally felt the need for and seek it. This sort of information is valuable, if the receiver is ready for it. So I wouldn't necessarily preach these lessons on my younger self or any younger individual. Instead I would do what I'm doing today, leaving it in an accessible place for curious minds to discover.

1- Approach to Mastery and Learning

The real difference between the master and the apprentice is the amount of time they have failed.

There is only one method for mastering any aspect of your life, that is practicing. Everything easy now was difficult some time ago, only thing changed in between was repetition.

Being smart only dictates how fast you can learn something new. How fast you can grasp something, and there are also different types of smarts, you need to learn each and respect every one of them.

Expose yourself to new information, be curious and adventurous. You can only learn what you don’t know if you expose yourself. You can also never know if a piece of information can benefit you, don’t waste time trying to learn everything but don't see your time wasted when you’ve learned something you don't see an immediate benefit.

Learn the gist of things, the main idea, the core principles of everything you do. There is not a tutorial for everything so you need to make your own decisions. To figure out complicated looking questions, you need to have principles you can fall back onto simple directional guidance that will help you figure out the answers.

In order to achieve this, you need to focus on learning problems and solutions instead of tools. If you know what problem you are solving in the essence and what solution you would like to reach, you can achieve anything in between. If you are lost in between these, then you would only become an operator of tools.

2- Understand Seniority in Business

No matter the circumstances, some things just happen with time. Experience is one of them, experience can only come with time because there is not a real methodical way to gain it.
Some things you can’t learn by reading a book, it’s true. You need to deal with people, customers, deadlines, mistakes, conflicts, headaches, heartaches and a lot more professional and personal drama. They all shape a person’s experience. This is not something that I’ve immediately acknowledged during my initial years. Sometimes, just being in the room long enough, is valuable enough for a business.

Also, do not forget that no matter what you’ve lived so far, your perspective is still only your perspective, valuing other people's opinions, at least giving them a fair assessment, increases your perspective exponentially.

Ask right questions and you will get excellent support from any senior in any field. However, as time goes by time becomes more valuable. So you need to know how to approach them, and how to make best out of their time in an efficient way.

3- Don't Shy Away From a Challenge

There is something brilliant about being daring

Awkward situations.You can actually tell how much you are getting out of your comfort zone by the amount of times you find yourself in situations where you don’t exactly know what to do. Every time you feel awkward, know that you are actually doing well. Obviously the exact same scenario shouldn't make you feel awkward in the long run but initially it will, and again it's something like after workout muscle pain, that's how some people feel more 'trained'.

It's similar to anything in life. If your job becomes too easy, you need to do something harder, much like you would go to higher weights in the gym. You also can't stop challenging yourself. Even without increasing the intensity, you can have diversified challenges.

An example is, while working on my technical skills in DevOps, I also take every other opportunity I can find to participate in sharing, teaching and mentoring in business and my personal life, because I want to give public speeches, write books and maybe run workshops in the future.

I take on projects, where the technology is something I'm already very good with, but took on the project just to improve on my project management skills. Again, If you want to be a leader, you need to start leading things, you need to accept delegation and actually own and deliver and deliver good results to the people who have trusted you.

There is no denying that I'm ambitious, and this ambition provides me a different perspective to spot opportunities otherwise may seem not really glamorous, or seem like a burden. I see a room full of growth opportunities.

Yes it's difficult to do extra, but easy is not our friend.

Easy Paths Leads to Crowded Destinations

Image source: https://m.facebook.com/visualizevalue/photos/a.749620378758293/749620342091630/?type=3&source=44

Check out Jack Butcher’s other work: https://twitter.com/jackbutcher

4- Don't Whinge

And own your destiny end to end

20s are difficult, I get it. Transitioning into being an adult is difficult, yes. However the earlier you accept the facts the less time you would lose complaining. Life is no war movie, there are no reinforcements, no cavalry will come for you. You are responsible for your outcome %100.

Complaining is just time lost, period. When you are a child, you can complain to your parents, if they allow you. It ends there. Unfortunately, this is a behaviour we bring from our childhood to our 20s, the earlier you snap out of it the better it is for your own sake.

I’m old enough to know that chaos will happen, uncomfortable events will happen, my only chance is to separate my decisions from my reactions. This is only possible, if you accept and internalize the fact that you have no one to complain to. This is different from asking for help, ask for help but when things go south, accept your responsibility. You are far away from perfection like everybody else in the world. Just try to learn from them, and visualize the alternative. If you never experience negative events, you would be crushed on the first occasion.

If you know that, complaining is not an option, as a reflex you can block your emotions. The better you can do this, the more that challenge transforms into a learning opportunity.

5- Funny How Ideas Change So Quickly

Even throughout the last 10 years, my ideas changed on so many topics I can’t even list them.
Be open to being wrong and be ready to update yourself. Your current views are shaped with your current experience, so don’t be a fanatic about them. Know that they are most likely to be changed in the future via various inputs to your life.

6- Be Authentic

Being authentic is effortless and It provides mental clarity.
I believe one of the reasons I did not feel the famous “Imposter Syndrome” is that I was, maybe sometimes too honest with my employers.

I was very open with what I’m currently capable of and my potential during all of my job interviews. If I was inexperienced in an area, I would tell them how much time I would require to do some up skilling on it.
This behaviour may have cost me some opportunities, but on the other hand I always had the confidence that I was very open to my employer, what I can deliver to them at the present and I am capable of picking new things as well.

7- Success Is Just Putting In The Work

The success you have achieved, won't feel like success in the beginning not even in the middle.
It will feel like you are chasing away a freaking train, if you look all the way ahead to the result.

The champions, %99 of the time, are not doing glamorous things, they just wake up and train hard. They have no fans cheering in the gym, there is no press, but they know they need to put in this work to actually perform great in that %1 of the time when it matters.

Success looks like getting sweaty in a gym, not like having an after party in a rooftop bar. That may be a byproduct, but it won’t feel like that until you get there. Whatever your goal is.

Between my 20s and today, my curiosity or growth mentality didn't change, one bit. What has changed is execution. I realized that, unless you take action and do work, information is useless.

This was possible when initially I took a chance on this theory that claims success is more about consistency and repetition of ordinary effort rather than bursts of extraordinary effort.
Actually sticking to doing things even if I don’t feel like it, and see that it’s actually working.

You can only see your success, either if people tell you so, or you really objectively take a step back, zoom out and compare yourself to your past self. In the present though, if you are growing you will always have that ‘itch’ and the feeling will suck. You will feel some sort of hunger, again like anything we are human and this is the human reaction. We want instant gratification, accepting it's not possible and that is the natural way is important for staying resilient.

8- Ideal World Does Not Exist

Ideal job, does not exist, Ideal project does not exist, Ideal working environment does not exist.
There is fine balance between having standards and expecting everything to be ideal.

If you feel the world is coming down every time you encounter something different than ideal, you would be exhausted. Accept that there would be adjustments in your plans, almost all the time.

Learn to lower your expectations just a notch, to protect yourself from emotional let downs. If you have it better, good for you, else well it’s fine we already settled on that it won’t be easy right.

9- Technical Debt is Not Your Personal Fault

When you work in a company and you see there are things that don't sit right with you, be open about it and speak up but don't get stressed. Getting fresh out of learning, you may think that in the commercial world, everything would be a textbook. On the other hand it’s not always the case, purely because we live in a capitalist economy, and business needs to consider cashflow, period.

Best practices are known and obvious but not always feasible. And drift doesn't necessarily happen due to people's talent shortage, ignorance or anything like that. it's simply a business decision, to prioritize something else and we are not the judge of this decision.

The judge of that decision is the management of that particular business, we are responsible to inform others about the consequences of the shift, but knowing why it happens may put your mind into ease.

Actually, your manager, the ex-developer, the ex-product manager most likely had an idea what would be close to the best practice in the first place, or they may have already realized a certain thing needs refactoring. However, business doesn’t work that way, maybe they were understaffed, or working towards a tight deadline for a feature. They may have simply not had the go ahead to go back to fixing things.

If things gets out of hand, you have every right to raise concerns, but don’t expect to not have any clunky processes leftover in any environment.

10- Understand the Business

What exactly are you getting paid for? What is the value you are providing? What problems are you solving? How does your work translate it’s outcome to your employer/client? How do you reduce their risks/costs?

Figuring out the answers to above questions is necessary because not all companies are providing clear success metrics to their employees. You may be lucky if this is more methodically approached by HR specialists but even in the absence of that guidance, be aware of your own position.

In a business context:
You are %100 responsible for your interests.
You are liable to a business, you are professionally contracted to.

These two are everything you need to take care of, you can't fall behind on your obligations to the business, that you are bound with a contract. It can be a direct, employee/employer relationship or it may be a direct customer of your services. It doesn't change the fact that, you can't fall short on these, this is your core integrity as a professional. Always keep this intent first.

On other other hand you are %100 responsible for the reward or compensation you are getting in return for your value. These interests, perks or monetary valuation is something you are responsible to gain.

Think about what your purpose is, when you are at work? Is it simply a paycheck, I don't think so right, you want to do more. Does work provide you the opportunity fulfil your dreams, if not have you asked if things might change or not? if still not why are you still there?
It’s a two way street. You need to be skillful to be desired in a business context, however the job you do needs to be challenging and rewarding enough to keep you growing and motivated.

Business is purely a value exchange, what do you solve for your client/employer, what do they then solve for their customers? How much better you made their ability to serve their customers is your value?

Once you figure out all above, you then have more clear vision on your self-worth in business.

11- Your Best Work Will Come Out Of Your Centered Self

There will be stressful times, where “shit hits the fan”. There will be downtimes, production issues. You will be tired, you will be mad, you will be frustrated.

Going back to not complaining philosophy, I would also suggest blaming and pointing fingers is not an attractive skill to have in any area of life.

What will make you a superstar is keeping your composure and trying to block off your emotions. Just think of solutions with your rational mind. By the way, solutions might be asking for help, however make all decisions with a centered peaceful mind.

Rushing, and acting on things trying out different things without any particular good reasoning is not such a good idea. In any business context, keeping your composure under chaotic times is a desirable and useful skill to have around.

12- It is Your Responsibility to Stand Out

Everyone is super busy, and has their own lives and responsibilities. People don't really have time to look out for your progress. You need to be vocal about it and keep track of your own accomplishments.

Don’t expect the road will also be carved out for you. You are again responsible for aligning yourself to what is next for you in your life, career wise. You need to figure out the next direction and walk and leap towards that direction.

Don’t expect people to give you credit just like that, that won’t happen. You need to be daring and actually try to stand out. It can only happen if you try and improve.

There you go, 12 pieces of advice I would give to any 20 year old today who would be entering the IT world today.

The insights above are all part of my belief system, but as like any teacher I’m also imperfect and I can’t always practice them as perfectly as I wish to be. I’ve titled this article as learnt, as I have a better understanding and solidified views about the topics above in comparison to my younger self, however I’m also a firm believer that learning never ends.

Hence, anyone who is reading this, the above insights are my experience distilled from 11 years of industry and 31 years of life experience. We are all students in life and can all learn from each other in some way. Be good.

Azure Architecture Scenario: Protect an Azure App Service with a Cloud Hosted WAF (DNS Based)

MertSenel — Sun, 23 Aug 2020 07:53:35 +0000

A sample architecture that shows how to restrict access to an Azure App Service from a Cloud WAF provider's IP Ranges

Project Overview

This is a whiteboard sample architecture that shows how to secure access to an Azure App Service hosted public and client facing application/api behind a WAF that is managed and hosted outside.

This particular example was based on Incapsula/Imperva Web Application Firewall and their implementation but it would be very similar for most of the DNS based WAF providers.

As the main requirements doesn't change:

First, you need to proxy(pass) the traffic coming to your application via your WAF provider's infrastructure, that way you make sure that the traffic is inspected by the WAF and rules you defined are applied to it.

Second, you need to make sure your WAF can reach your origin servers

and Third, make sure you are limiting access to your App Service only from the Web Application Firewall's Outbound IP ranges.

This flow would be similar for a Virtual Network hosted applications as well such, as Kubernetes Clusters, Virtual Machines, App Service Environment in short anything where you can apply a Network Security Group. If this is the case one can just apply the same whitelisting logic of the outbound IP ranges of the WAF provider, to the inbound rules of the Network Security group.

The reason I've made this sample with the regular Public App Service is that, for someone who doesn't knows it's full feature set, they might consider it an hosting solution that can't be secured or locked down. They might say, it's public hence it's not secure think they need to either opt-out from using App Services or forced to upgrade to an App Service Environment.

If you use another provider like Cloudflare where they also forces you do use them as your DNS provider, in this case the CNAME record is automatically handled for you, you just need to make sure you turn on the "orange" cloudflare proxy mode on for your particular subdomain and limit access to your App Service to only Cloudflare's outbound IP ranges. I had setup something very similar in the past with Cloudflare WAF and with Azure VM and Load Balancer.

Diagram Details

1- Normal traffic from the users of our application. Their traffic will be processed by Web Application Firewall's engine and expected to be allowed.

2- Malicious Attacks made to our application will be picked up by the WAF rules processing and blocked.

3- All Access to the App Service outside of the Cloud WAF Provider's Outbound IP range is blocked by the App Service Access Restrictions.

To view the diagram in draw.io viewer please click here.

Azure ARM Template Development - Validate with WhatIf

MertSenel — Sun, 16 Aug 2020 06:24:36 +0000

Hi, I'm back with yet another script I would like to introduce and pretty much document for my and other's future reference.

Today's script is named "Deploy-AzARMTemplate.ps1" and it's geared towards one-click deploys from your local development environment to pre-configured cloud dev/test environment. Making use of the new What-If validation the script also allows us to pre-validate the deployment.

Purpose

So what does this script does? or what is it for? This script's main purpose is to allow DevOps engineers to be able to make a full deployment to a pre-set environment for testing purposes. The script is easily modifiable and can work with multiple projects, given they are using similar naming structure.

The benefit of this portable and configurable script is to reduce the time spent on curating different AZResourceGroupDeployment cmdlets,
such as Test-AzResourceGroupDeployment , Get-AzResourceGroupDeploymentWhatIfResult and obviously the real deal New-AzResourceGroupDeployment which generates the deployment.

I was aiming to set the parameters once and then just run the script by itself or with some easy common switches such as -Test , -WhatIf or -Force to be able to perform these functions.

This script is aimed to be used locally while developing/extending your ARM templates and allow for quick deployment to a Dev/Test environment. Hence keeping your Azure Environment information is not insecure as long as your repository is private and only people who you want access has access to it. Azure Subscription ID or Tenant IDs are not exactly secrets on their own, but it may expose you for some vulnerabilities if they are known together by a 3rd party.

Prerequisites

Make sure you are on Powershell AzModule version 4.2 or higher. The script already enforces this.

To learn more about the new WhatIf operation you can read more here: https://docs.microsoft.com/en-us/azure/azure-resource-manager/templates/template-deploy-what-if?tabs=azure-powershell

The Script

Without further do let's get into it.

I'll break down the Script into sections first.

First let set our parameters block and configure the path where our ARM template files resides and also Azure Environment Information

#Requires -Module @{ ModuleName = 'Az'; ModuleVersion = '4.2' }

[CmdletBinding()]
param (
    [Parameter()][switch]$WhatIf,
    [Parameter()][switch]$Test,
    [Parameter()][switch]$Force
)
#region Configuration
#Deployment Assets Configuration
# Go Up one level so you can pick a project folder.
# This is here in case you want to keep your script in different folder.
$OperationsPath = Split-Path $PSScriptRoot
# Target the folder which desired ARM Template and Parameter files are in.
$ArmArtifactsPath = "$OperationsPath\infrastructure"

#Azure Environment Configuration
$TenantId = '#{YOUR-AZUREAD-TENANT-ID}#'
$SubscriptionId = '#{YOUR-AZURE-SUBSCRIPTION-ID}#'
#endregion

Then we make sure we are in the correct Azure Context

#region Connect to Correct Tenant and Subscription
$CurrentContext = Get-AzContext
#If there is no AzContext found connect to desired Subscription and Tenant
if (!$CurrentContext) {
    Connect-AzAccount -Tenant $TenantId -Subscription $SubscriptionId -UseDeviceAuthentication
    $CurrentContext = Get-AzContext
}
#If subscription ID doesnt match, call the Set-AzContext with
#SubscriptionId and TenantId to allow switch between tenants as well.
if ($CurrentContext.Subscription.Id -ne $SubscriptionId) {
    $CurrentContext = Set-AzContext -Subscription $SubscriptionId -Tenant $TenantId
}
#endregion

Then let's configure the service parameters. These sections are all seperated so everyone can just configure that section.

For example, here use a ResourceGroup name structure where I just purely concatenate the components but you may be using a whole other,
structure like having '-' in between those parts.

The idea is, we have all the different identifier enums about our service so we can curate whatever string we want use.

#region Service Constants
$ProjectName = 'mert'
$Stage = 'dev'
$ServiceType = 'fnc'
$Region = "wus"
#endregion

#region Service Curated Parameters
$ResourceGroupName = $ProjectName + $Stage + $ServiceType + $Region + '01'
$TemplateFile = "$ArmArtifactsPath\template.json"
$TemplateParameterFile = "$ArmArtifactsPath\parameters.$Stage.$($Region)01.json"

$ARGS = @{
    ResourceGroupName     = $ResourceGroupName
    TemplateFile          = $TemplateFile
    TemplateParameterFile = $TemplateParameterFile
    Mode                  = 'Incremental'
    Verbose               = $true
    ErrorAction           = 'Stop'
}
#endregion

Deployment Helpers, to only run certain mode. They are both good for certain things.
The Test-AzResourceGroupDeployment performs a very primitive check and it's already covered in regular or what-if deployments but it is still somewhat useful.
These can sometimes detect problems that VsCode ARM Extension cant, so it's here to make the script complete but using this mode completely optional.

Get-AzResourceGroupDeploymentWhatIfResult is something I use a whole lot more nowadays as it attempts to simulate the real REST API response from the Azure side, assuming the deployment was made.

As you would see in Microsoft's disclaimer as well that it's still in preview mode so it may still generate some extra noise.
So practicing with it now, when it's out of preview, you will be ready to use it to it's full extent!

#region Deployment Helpers
$ErrorActionPreference = "Stop"
if ($Test) {
    Test-AzResourceGroupDeployment @ARGS
    exit $LASTEXITCODE
}
if ($WhatIf) {
    $WhatIfResult = Get-AzResourceGroupDeploymentWhatIfResult @ARGS `
                                    -ResultFormat FullResourcePayloads
    $WhatIfResult
    exit $LASTEXITCODE
}
#endregion

Finally the part that performs a WhatIf first and deploys.
This is pretty useful as you pretty much be able to see a Delta of your changes.

If -Force flag is passed to the script then, the -Confirm flag is set to false hence the deployment will just proceed without confirmation.

This is useful if you know you already know that your template is valid, healthy and deploys fine but you are just making changes to it and want fast deployments.

#region Deployment ("Validate & Deploy" or "Forced" Deployment)
try {
    $PromptForConfirmation = (($Force) ? $false : $true)
    Write-Host "Deploying $Region"
    $Deployment = New-AzResourceGroupDeployment @ARGS `
                                                -Name "$(New-Guid)" `
                                                -Confirm:$PromptForConfirmation `
                                                -WhatIfResultFormat FullResourcePayloads
}
catch {
    Write-Host $_.Exception.Message
    exit $LASTEXITCODE
}

# Tell me If my Deployment was Successfull
if ($Deployment.ProvisioningState -eq "Succeeded") {
    Write-Host "Deployed $Region Successfully"
}
# Or if there was an error during Deployment I want to know about it
elseif ($Deployment.ProvisioningState -ne "Succeeded" -and ($Deployment.CorrelationId)) {
    Write-Host "$(Get-AzLog -CorrelationId $Deployment.CorrelationId)"
}
# Else just let me know if there was no AzResourceGroupDeployment Object found
elseif (!($Deployment)) {
    Write-Warning "No AzResourceGroupDeployment Object Found"
}
#endregion

In this section, we pretty much use the -Confirm switch built-in to the New-AzResourceGroupDeployment cmdlet to achieve the prompt behaviour.

The rest of the code is for catching exceptions for broken/invalid template deployment(or WhatIf) attempts and print the error/exception message as needed.

What is Cool About WhatIf

The WhatIf check makes ARM templates look like terraform plan command that tells you what would change.

However, terraform works with a state file, here the state file is the actual state's of the Azure Resources, being retrieved upon request via the REST API.

Terraform, populates the state file itself so when you do a deployment after the first time, the state file is compared to declared deployment. This is a very accurate difference as both are generated by Terraform.

You need to use the deployment mode "Complete" instead of the default "Incremental" if you wish this new WhatIf check in ARM Template deployments works closer to Terraform.

But because you are comparing the actual live resources to your ARM template hence it's not easy to always 1:1 match the default generated properties to your ARM template, or there are some properties which are just metadata.

Hence currently there is noise in the output but that doesn't make it any less of a tool to be used for Testing your ARM template if it will pass an actual deployment request made to the Azure ARM Rest API endpoint.

Or to simply visualize and see the impact printed out for you after you have made some changes, updating or extending your ARM template.

The full script file can be found at: https://github.com/MertSenel/devops-powershell/blob/main/scripts/local/arm-template-development-helper/Deploy-AzARMTemplate.ps1

Script in Action

Folder Structure

I use a folder structure like below, I'm sharing this to give the script some more context, in terms of how the files are named.

Sample Folder Structure

Test Mode

.\Deploy-AzARMTemplate.ps1 -Test

Test Mode

WhatIf Mode

.\Deploy-AzARMTemplate.ps1 -WhatIf

What If Mode 1

What If Mode 2

Deployment Mode

.\Deploy-AzARMTemplate.ps1

Performs WhatIf and presents the changes before proceeding with the deployment.

Deployment Mode - Pre-Deployment Confirmation

if "Y" is selected proceed with the deployment

Deployment Mode - Proceed with Deployment

if "N" is selected deployment is aborted.

Deployment Mode - Abort the Deployment

Forced Deployment Mode

.\Deploy-AzARMTemplate.ps1 -Force

It will do the same as above but, won't ask for confirmation. Just runs WhatIf Validation and if no error thrown, proceeds with the deployment.

You can think of this like terraform apply -auto-approve command where it skips the interactive approval.

Conclusion

This is very handy script to move around with you in your different project and configure it once to your desired ARM Template project and you can always deploy it to your desired dev/test environment with ease.

This script can also be extend to accept some sensitive parameter values via adding a securestring parameter to the script and also extending the Hashtable.
This is forward the secret value to your ARM Template deployment as input in a secure manner, so you dont have to save it in plain text.

You can use this to pass any type of secrets to an ARM template.

param (
    [Parameter()][switch]$WhatIf,
    [Parameter()][switch]$Test,
    [Parameter()][switch]$Force,
    [Parameter()][securestring]$mySecretValue,
)

$ARGS = @{
    ResourceGroupName     = $ResourceGroupName
    TemplateFile          = $TemplateFile
    TemplateParameterFile = $TemplateParameterFile
    Mode                  = 'Incremental'
    Verbose               = $true
    ErrorAction           = 'Stop'
    mySecretParameterName = $mySecretValue
}

You can also convert any of the constant to an array and deploy to let say multiple regions with single loop and convert this script to make more than one deployments.
It's flexible after this point.

#region Service Constants
$ProjectName = 'mert'
$Stage = 'dev'
$ServiceType = 'fnc'
#$Region = "wus"
$Regions = "wus", "aue" ,"neu"

foreach($Region in $Regions) {
    # The rest of the Deployment code logic goes here.
}
#endregion

Another benefit is that, having this source controlled and not testing your deployments via your terminal will also give you more confidence in the configuration you are passing to your deployment.

Last benefit is, you can add breakpoints to your script to debug a certain value at any given time if you are having troubles getting your deployment working.

As always I hope this script helped you out and let me know if you have any feedback or criticism about it via comments section.