DEV Community: stephenatwell

Building Trust in Your Deployment Tools

stephenatwell — Tue, 26 Apr 2022 22:22:06 +0000

The Cloud Native Computing Foundation found an extraordinary 300 percent increase in container use between 2016 and 2020. Kubernetes’ popularity is steadily rising, with 83 percent of respondents using it for production as of mid-2020.

Kubernetes’s out-of-the-box capabilities enable organizations to accelerate delivery cycles and rapidly scale their operation, contributing to its increasing adoption. However, organizations must support the container orchestration engine with secure and efficient release and deployment workflows to get the most out of it. The CNCF 2020 report highlights container release and deployment complexity among the top challenges the respondents faced.

DevOps engineers deploy to Kubernetes (K8s) by updating associated YAML files. While it may look easy for a single deployment, it quickly becomes arduous when performing multiple deployments across various environments. These include things like development, quality assurance (QA), user acceptance testing (UAT), staging, and production.

Although Kubernetes supports rolling updates, it’s challenging to do it manually since a Kubernetes application requires multiple YAML files. Manually building these files can be error-prone because of the white space syntax.

Also, handcrafting files requires substantial knowledge of Kubernetes components. These components can complicate the overall deployment process when you need to upgrade some to get the application working.

Auditing upgrades to track new and old deployments is also challenging to do manually, as is managing rollbacks for applications comprising several deployed services. Secrets management also becomes challenging when teams work on a distributed app across various repositories. Yet, containers must be secure to be trustworthy.

Many customers also desire more advanced deployment strategies than rolling updates in order to decrease the blast radius of a given change. Leveraging advanced strategies such as Canary Deployments and blue/green deployments ensures that if a new software version has an undetected issue, it affects as few users as possible.

Some of these challenges are unique to Kubernetes, but the other, broader issues apply to successful software delivery. Adopting continuous integration and continuous deployment (CI/CD) practices can address these challenges. Using tools to automate the entire deployment process reduces Kubernetes management effort so organizations can focus on core business needs.

Let’s explore how some of Armory’s automated tools help with CI/CD and address Kubernetes scaling and security concerns.

How Spinnaker and Armory Enterprise help with Kubernetes Scaling and Security

Self-serving enterprise development teams use continuous integration and continuous deployment (CI/CD) to deliver applications faster. These CI/CD practices enable business agility to give organizations a competitive advantage. CI/CD also leaves more time for innovation within your organization.

Rolling out applications to Kubernetes requires CI/CD solutions aimed at solving deployment challenges. Several open-source solutions help automate continuous delivery pipelines to Kubernetes, like Spinnaker, ArgoCD, and Flux CD. These solutions treat Kubernetes applications as first-class citizens, deploying the applications across various Kubernetes platforms, including Google Kubernetes Engine (GKE) and Amazon Elastic Kubernetes Service (EKS).

These CI/CD tools give organizations agility and control. They can implement various deployment strategies and perform rollbacks with all the required release governance.

Armory agent provides simplified and scalable management for all of your Kubernetes clusters when using Armory Enterprise or Open Source Spinnaker to manage your deployment pipeline.

GitOps

Armory Enterprise’s pipelines as code feature allows you to maintain your Spinnaker pipelines within git, supporting a GitOps strategy. GitOps leverages Git as the single source of truth and control mechanism for creating, updating, and deleting system architecture. First, Git provides permissions control, ensuring not just anyone can make changes. Second, because Git makes all the changes, organizations can define the infrastructure immutably to prevent other changes.

Organizations Face Kubernetes Scaling Issues

Open-source CI/CD solutions work well when organizations are early adopters, with small cluster sizes and a few deployed applications. However, they aren’t well-suited to large clusters.

For one, when a cluster contains many applications, the user can experience a delayed response to their action, and deploying changes becomes painfully slow. When the service is slow to respond to invocations by Kubernetes cluster components like kubelets, registries, API servers, and more, it causes timeouts and failed deployments.

Dashboards also often fail to capture deployment changes in near real-time and display stale information. The refresh slows as the application purges all the data.

Finally, these open-source solutions are not suited to scaling individual components to handle many clusters from the centralized instance. It’s challenging to understand the working details of the open-source solution.

One open-source CI/CD solution, Spinnaker, has an enterprise version that addresses these scaling challenges. Armory Enterprise, which is built on top of Spinnaker, works on clusters of any size, addressing scalability issues through improved redundancy. Its complementary features like infrastructure dashboards, drift monitoring, role-based access control (RBAC), audits, and more fulfill the development team’s needs. Armory also offers its Agent as a standalone offering for Open Source Spinnaker that increases its ability to scale to large-scale Kubernetes infrastructure.

Open-Source Tools Trigger Security Concerns

As open-source CI/CD pipelines automate and improve the software delivery workflow, they pose significant security risks. You need to trust that these solutions are not making your organization vulnerable to attack. Let’s examine some of these risks then see how an enterprise-level solution helps.

As enterprises prioritize software delivery speed, security and compliance are sometimes an afterthought. However, to trust their deployment tools, organizations must actively integrate security and compliance throughout their deployment pipeline. Their ability to adopt specific cloud-native tools depends on their current IT topology and organizational maturity.

Adopting a centralized CI/CD solution presents security concerns that require rigorous monitoring, command, control, and policy mechanisms. The CI/CD solution is a single source with access to all Kubernetes objects in all clusters. It often runs with full-access service accounts that can provision new components, stop existing services, read secrets, access external components including other clusters, and more. Additionally, it persists information to perform various analyses and generate dashboards.

Enterprise-level solutions like Armory Policy Engine can help centrally enforce policy on your CI/CD pipeline to improve security and make deployment more trustworthy. Additionally Armory Agent allows central management of Kubernetes accounts, helping you avoid proliferation of credentials that can access production.

Using Armory Agent for Trustworthy Deployment

Armory Agent is a departure from the centralized CI/CD solutions. It builds upon Spinnaker’s capabilities and makes the deployment process trustworthy at a massive scale — even when you have thousands of Kubernetes clusters.

The Agent is a lightweight, scalable service you deploy in a Kubernetes cluster. Each agent manages a single cluster so that you can scale the distributed approach for any number of clusters.

The embedded agent monitors all infrastructure and streams changes back to Spinnaker’s Clouddriver service. The distributed approach delivers several security benefits.

First, the Agent’s real-time sync captures the latest deployed changes. Also, since Spinnaker Clouddriver interacts with the agent, it decouples direct access to the Kubernetes API server, limiting a bad actor’s potential access.

Instead of centralizing account management, storing all Kubernetes accounts and credentials within Spinnaker, the Agent manages service account credentials in each cluster. This approach improves security by limiting teams to the clusters they manage.

Teams can manage permissions specifically for their clusters and quickly add new accounts without restarting Spinnaker. Finally, organizations can keep K8s clusters private with limited access since only the information Armory Enterprise requires leaves the cluster.

The Agent mitigates performance and latency issues associated with a centralized Spinnaker. It optimizes how infrastructure information is cached, making force cache refresh almost instantaneous. End-users benefit from optimal application performance, and DevOps teams benefit from optimal pipeline executions.

How Armory Project Borealis Improves Trust

Armory also offers an early access SaaS deployment offering called project borealis. It is a turnkey offering that incorporates many of our solutions from Armory Enterprise.

Project Borealis allows centralized management of Kubernetes clusters at scale. This allows you to control credential proliferation and manage all your clusters with a single toolchain. Project Borealis can also help you orchestrate your entire deployment lifecycle, across multiple environments, hosted across many different Kubernetes clusters. This helps ensure that your deployment process is consistently followed, decreasing risk and improving trust in your tools.

Project Borealis also provides advanced deployment strategies including Blue/Green Deployments and Canary Deployments. These strategies allow you to minimize the blast radius of your deployments, so that if the software contains an undetected issue the user impact, and any negative impact on trust, are minimized.

Project Borealis is currently available as an Early Access offering to our design partners. Spaces are still available, but filling up fast–so sign up today.

Summary

Kubernetes and cloud-native applications unlock business agility, but only when organizations support them with Kubernetes-specific CI/CD tools. Processes requiring manual intervention are error-prone and often cause bottlenecks. Organizations need automation to get the most out of their containerized services. They also need enterprise-level tools to trust that their deployments are secure.

Armory provides the secure, scalable, efficient, and reliable continuous delivery (CD) platform that you need. It builds on Spinnaker with policy-driven deployments that enable you to innovate with speed and safety.

Contact Armory today for a complimentary assessment of your software delivery practices and learn more about how your organization can benefit from safe, reliable deployments.

The post Building Trust in Your Deployment Tools appeared first on Armory.

How Progressive Delivery Drives Fast Feedback and Insights

stephenatwell — Mon, 25 Apr 2022 16:17:46 +0000

Software has grown exponentially across all industries in the last decade, both in adoption and complexity. This growth is due in part to digitization across sectors. It’s also in part because of the constant evolution of software development practices, which enable developers to build more exciting software better and faster.

Agile methodology has become the de facto standard for software development. Developers satisfy customers through early and continuous delivery of valuable software, then accommodate changing requirements throughout the development process.

Many companies have adopted some form of continuous delivery (CD) to reduce the feedback cycle, improving their software quality and development speed by steadily fixing bugs and adding features. Customers quickly get their requested changes instead of waiting months until the next software version.

We’ll explore how continuous delivery and progressive delivery enable a continuous feedback loop to improve software quality then discuss how Armory helps.

Continuous Delivery

Continuous delivery releases code changes automatically, either internally or externally. This makes a deployment repeatable, reliable, and predictable. When deployments are easy and reliable, it frees your developers to work on features instead of troubleshooting the deployment. They can then take the time to follow best practices instead of cutting corners on the deployment configuration.

So how can you use modern continuous delivery practices to improve software quality and delivery speed? By deploying often.

A modern application typically contains a few different components, each hosted on multiple machines behind a load balancer.

When your engineers release an application, they must deploy the code to not just one server, but all servers hosting the services. This method is both tedious and error-prone. Tools like Kubernetes help automate this process, enabling CD pipelines to push changes to the entire cluster using one simple command.

The ability to automatically push changes reduces your software development cycle’s feedback interval, increasing your software quality and delivery speed.

Progressive Delivery Drives Continuous Feedback

Progressive delivery builds on continuous delivery with two main additions: the use of feature flags to test code in production and progressive rollouts.

Many techniques have evolved to facilitate a rapid feedback cycle. Modern techniques like canary and blue-green deployments progressively release the application. These methods deliver the changes to only a subset of users, making it possible to safely test new features and gather data in a real production environment without a huge impact.

With progressive delivery, the development and release cycle typically looks like this:

Teams develop code, commit it, and prepare it for release.
They release the code to selected targets.
Companies gather relevant data through user feedback, telemetry, or statistics, and use the information to plan the next round of app improvements.
The cycle repeats.

One possible release strategy is to push the release to only internal stakeholders. Teams release the version to a broader public when internal stakeholders are satisfied with the changes. Customers won’t see the features stakeholders aren’t satisfied with in the release.

To optimize learning, progressive deployment can push new designs to a subset of real users and gather real usage data from a live production environment.

With the help of an observability tool, you can gather and aggregate all kinds of data relevant to your application, including logs, events, performance, and errors. Such tools collect data and also analyze the information to help developers resolve issues faster.

Now all the stakeholders have reliable and indisputable data to learn from. For example, your development team may have a solution to a performance problem, and they want to test it out by deploying to just 10 percent of the traffic through canary releases. As performance and usage data comes in, your team can roll out to more users and observe when performance degradation happens using real load. It’s usually challenging to simulate load using internal tools but possible with progressive delivery.

Iterative development is the engine of a rapid continuous feedback loop that cycles between development, feedback, improvement, and delivery.

Using a progressive delivery strategy can minimize downtime for each release. After-hours or midnight deployments are no longer necessary. Customers are happy, your release engineers are at peace, and your product analysts have the data they need to drive the next round of improvements.

Armory Enables CD and Progressive Delivery

Thanks to tools like Armory, developers can confidently release their changes to production and help stakeholders gather the information they need to drive the next round of iteration. Armory is an enterprise-grade version of Spinnaker, a continuous delivery tool built on best practices learned by delivering software at scale.

The software has many built-in features for you to implement progressive delivery with confidence:

Canary analysis is available out of the box. It’s easy to create a canary image before each deployment, collect some data, then either manually decide to push the release or make the decision automatically within a pipeline.
Once the application is released, you can undo the release through a pipeline or with a manual button click.
You can scale Kubernetes deployments. When the number of clusters grows, it’s typical to experience latency issues, and managing configuration and credentials also becomes a logistical nightmare. The Armory agent for Kubernetes addresses both concerns with a decentralized model and scales to thousands of clusters with ease.
Its policy engine ensures every rollout meets a set of codified standards. By setting appropriate policies, you can ensure every deployment is compliant with your organization’s standards, unifying DevOps and SecOps in the release process.

While you can implement progressive deployment with other tools like Kubernetes, Armory enables continuous delivery at scale.

For companies seeking a SaaS deployment solution, Armory also offers Project Borealis. Project Borealis delivers intelligent deployments-as-a-service so that your developers can focus on building great code, rather than deploying it. Project Borealis is currently available as an early access service to our design partners. Project Borealis is a drop in replacement for kubectl that can deploy using progressive deployments. If you’d like to learn more, sign up to become a design partner.

Conclusion

As software complexity has grown, the deployment process has become challenging. You can simplify this process with tools like Kubernetes to roll out changes to a cluster of servers at the click of a button, but that’s not all that continuous delivery is good for.

Today, progressive delivery powers many of the world’s best tech companies as the engine of a continuous learning cycle. Companies progressively roll out their product, gather accurate usage data in a real production environment, and use the information to help build their products.

Using Spinnaker as its core, Armory facilitates progressive delivery at scale, giving your developers the power to deploy with confidence. We invite you to check out what Armory can do for you.

The post How Progressive Delivery Drives Fast Feedback and Insights appeared first on Armory.

Reducing Risk by Automating Security with Continuous Delivery

stephenatwell — Thu, 09 Dec 2021 16:30:37 +0000

Risk avoidance has become a cardinal principle of modern enterprises. Companies like Armory have arisen to help modern organizations meet this principle.

In this article, we’ll highlight the risks that an organization faces with each deployment, how to test for them, and, ultimately, how to prevent them from happening. We’ll demonstrate how Armory can help minimize these risks by automating each deployment’s security and compliance checks. With Armory, your IT teams save money and time while outperforming human-based compliance.

Before discussing Armory’s risk avoidance process, let’s recap what most organizations do to tackle the challenges of compliance and security in their DevOps pipelines. Then, we’ll discuss how Armory can help companies smoothly perform all these checks automatically, with every deployment.

Understanding Risk in Deployments

Risks associated with deployment can come in different forms. Perhaps the Ops team did not properly handle configurations. Or, maybe deployments aren’t using secure vaults to store and provide secrets and API keys.

Your IT team should be able to control the behavior of your applications. This applies to runtime behavior as well. You can use feature flagging to control the behavior of your applications by enabling/disabling features at deploy time.

Additionally, you can mitigate risk through separation of duties. Use role-based access control (RBAC) to restrict certain application release tasks to specific roles.

You can also create a better disaster recovery plan so that your customers can continue to use your service even if there is a crash in the infrastructure.

Organizations generally hire experts in security, infrastructure, automation, and engineering to take care of these challenges. However, these teams are often undertrained to tackle these modern problems. Or, if they are sufficiently trained, they rely on legacy systems such as audit alerts and monitoring dashboards only. Also as each team focuses on its part of the problem, vulnerabilities appear in the overlap because neither team can address it on its own.

Methods like feature flagging and RBAC help you remediate modern problems and tackle the challenges they pose, but they slow down the overall deployment process. Even with a slow and cautious deployment process, the application will still not be compliant with some policies, and there is no guarantee that you can control your deployments. This is where an automated process can help your organization.

Automating Security

Modern organizations must adopt the use of automated security and compliance to stay competitive. Armory helps organizations stay on top of the challenges modern architectures pose — and solve them with a single solution that doesn’t require a large compliance or IT team. Our Policy Engine helps you solve all of these challenges with a single mechanism.

Most companies already have a compliance process in place, founded on maintaining specific controls. For example, there’s likely a division of responsibility. In this case, the person who writes the code isn’t the same person who deploys it. These division of responsibility controls may get in the way of software developers directly deploying and operating their own code, common in the DevOps revolution, and developers may try to get around this restriction.

Right now, companies usually rely on manually reviewing audit trail systems. A scheduled security scanner tool sends an alert when it detects an event that fails to meet the requirements of the control. The system then triggers an incident response standard operating procedure. However, all of this happens after the fact. With Policy Engine, you can prevent workers from ever deploying a noncompliant application.

Policy Engine also helps with some more traditional access controls. Similar to what a policy does in AWS IAM, it gives you control over access to resources. And, it can be finer-grained than the access controls in Spinnaker.

Using Policy Engine

Policy Engine helps organizations reduce the risk of security and data leaks in continuous deployments. It’s available from Armory Enterprise as a proprietary solution for organizations that want to control and regulate the entire software delivery pipeline. Policy Engine uses Open Policy Agent (OPA) and automation documents to write and run policy checks against deployments.

Policy Engine helps you in many ways. For example, you can add checkpoints to validate policies and compliance throughout your entire software development lifecycle. It provides example policies for security and compliance so you can get started quickly. Policy Engine also enforces policies like testing and verifying every deployment, preventing human error in the process. With Policy Engine RBAC, you can control who can do what in the pipeline, and with OPA’s decision logs, you can easily audit the changes made to the infrastructure.

Beginning to tackle security and risk reduction when developers are writing the code gives them an early start on securing the application. If any policy fails during the entire pipeline, Policy Engine prevents the deployment. Then, based on your business requirements, you can either prevent the deployment from reaching the production environment or reject the code push from the developer altogether.

The integration of Policy Engine is simple using the Policy Engine plugin for Armory Enterprise. This plugin quickly enables you to automate policy as part of your Spinnaker deployment pipeline. You can then write your own policies that control the flow of deployment. Automated policies ensure that your deployments are always compliant with the required regulations.

One thing your policies can control is ensuring secrets store your sensitive variables, such as API credentials, passwords, or connection strings. Policy Engine can enforce running automated tests to validate your deployments’ security. Additionally, Spinnaker can run specific tools such as OWASP scanners on your deployments as part of your pipelines. Policy Engine can ensure such scanners run and pass before deploying code to a production environment. Moreover, you can ensure the personnel involved in the deployment phase have been authorized to do so. You can check this using the role-based access control.

With these automated controls, your developers can push the changes to a staging environment, run scanners and automated tests, and have someone from the correct role approve changes before deploying to production. More than this, Policy Engine can connect to external services to request approved and blocked resources. Open Policy Agent (OPA) enables you to pull data, which helps you query external systems when validating the policy compliance.

Policy Engine not only checks the current state of your deployment but also helps you understand the history of your deployments to your production environment. Policy Engine stores the logs of every deployment whenever a policy runs and logs where the policy runs in OPA’s decision logs. You can query these logs using the API offered by OPA. This allows historic manual audit-based controls to still run while automating those manual processes and leveraging automatic policy enforcement.

Armory can also help you with disaster recovery (DR) on your infrastructure. Armory Spinnaker can switch your traffic from the active platform to a passive infrastructure in case of any disaster. These disasters could range from a solution crash to a data center failure. For an in-depth, practical approach to DR in Amazon Web Services (AWS) with Armory, review our disaster recovery documentation.

Next Steps

Organizations face many challenges with their software releases. DevOps and automated pipelines are not enough to ensure the security and compliance of your solutions. Some organizations try to solve these challenges by setting up teams and departments to manually prevent these risks in the system but fail to do so due to their slow pace and human error.

Armory helps you automate these risk evaluations to enforce compliance without slowing the pace of development. Using Armory Policy Engine, you can configure and set up policies for your automated deployments. To learn more about leveraging Armory in your environment, you can review our extensive documentation.

Now that you know how Armory can help you, you can request a demonstration tailored to your organization or set up Armory using Minnaker for a trial on your platform.

The post Reducing Risk by Automating Security with Continuous Delivery appeared first on Armory.

How to Take the Pain of Rollbacks out of Deployments

stephenatwell — Tue, 26 Oct 2021 16:26:21 +0000

Software applications have become an integral part of the business climate in most modern organizations. With an ever-increasing demand for new features and enhancement of already-existing ones, software teams constantly face novel challenges, and the pace of software development is growing by the day. To keep up with this fast-paced business climate, software teams must employ the right tools. This is where continuous delivery (CD) comes in.

Continuous delivery is a software development practice where the software team releases sustainable changes to users securely and in real-time. This practice is crucial to how software teams deliver value, and tech companies need it to thrive in the present times.

Continuous delivery follows a streamlined process generally referred to as the continuous delivery pipeline. At the beginning of the pipeline, the developer commits his code to the source repository. With each commit, automated tests and checks run to ensure that the code conforms to predefined standards. As soon as conformance is certified, the system automatically deploys the application to staging or user acceptance testing. Afterward, the committed code is ready for deployment on demand.

Despite all these checks, it’s still possible for the deployed code to contain bugs or inconsistencies. The only accurate method of fully certifying that the application or a new feature works according to specification is when a client uses it. A new commit may introduce a novel feature or fix a bug but break something else in some cases. The broken piece of code often goes undetected until it gets to the final user. The code is already in production and connected to many other important components, so minimizing downtime is essential.

Picture this scenario: One of your significant clients uses your application in the middle of the night to prepare for a critical presentation the following day. The client discovers that a feature needed to extract essential information is not working. They create an urgent support ticket, and you must attend to it immediately because millions of dollars are at stake. This isn’t the kind of situation any developer likes to be in, because it puts you and your competence in the spotlight.

A simple rollback to a previous commit should resolve the problem at least temporarily. This takes some of the pressure off you while you settle down to do a thorough investigation. But what if your tools make it difficult or impossible to roll back changes? Imagine the headache and embarrassment the downtime will cause.

Statistics show that only one in ten developers release multiple deployments each day. For about sixty percent of developers, a new commit takes up to a week to go into production. In many cases, it takes a lot of time to troubleshoot issues and restore service when an unplanned outage occurs. And even when it takes less time, developers often consider sacrificing stability, which leads to a cycle of frequent outages when something else breaks because of a previous fix. Employing the right tools can significantly improve the speed and efficiency of the entire CD process.

Avoiding Rollbacks

While rolling back changes can present a quick fix in many cases, it’s not the most optimal option for obvious reasons. Rolling back changes means that you have lost several hours of hard work and that your new functionality is not available. This is unfavorable both for the software team and the client. But then what if you could eliminate or at least significantly reduce the need for rollbacks altogether? Of course, it will lead to a significant leap in productivity and overall efficiency of the software development life cycle. However, this can only take place when you use a reliable continuous delivery tool. A tool that helps you avoid unnecessary rollbacks and makes the process seamless whenever you use it. Armory is the perfect tool for this purpose.

Armory, a premier member of the Continuous Delivery Foundation, offers an enterprise-grade version of Spinnaker — the world’s leading CD solution. This tool empowers developers to deploy instantly to the cloud. In addition, it enforces compliance, which fosters better security, and ensures that best practices are adhered to, minimizing the need for rollbacks.

Armory applies a policy-driven workflow to eliminate the need for manual deployment. It integrates information between developers and security operations teams. The Armory team creates custom policies with clients by working backward from a successful deployment based on the client’s needs and critical business objectives. When the security and compliance teams agree to these policies, Spinnaker automatically enforces them, providing valuable warnings and errors to developers so they can spot and resolve issues before they go into production.

Armory supports blue/green deployments (similar to red/black deployments). With the blue/green deployment technique, two production environments run simultaneously. One runs as a live server and the other as a test server, and you can easily interchange them. You test new features on the test server, and once certified, they are production-ready. The software team can re-route the workload from the live server to the test server. The team can switch back to the live server if they detect issues, reducing downtime to the barest minimum.

Armory also supports canary deployments where you deploy new changes to a small subset of servers for testing first. When you certify that the changes are viable, they cascade to the rest of the servers. This way a failed deployment only affects the canary servers, limiting the downtime users experience.

In addition to the blue/green and canary deployments, Armory offers policy enforcement, production architectural best practice enforcement, and production permissions and entitlements management. These tools employ various techniques to minimize the possibility of bugs in deployment and, in effect, the need for rollbacks altogether.

Automating Rollbacks

While developers strive to reduce the need for rollbacks to the barest minimum, there are times when it’s just the only way out — at least for temporary relief — while a development team tries to resolve an issue and restore service. When it comes down to this, employing the right tool is critical to ensure that the process is seamless and optimized for both the end-users and the software team. Armory automates rollbacks with one click, using features from Spinnaker. Spinnaker improves consistency and dependability of rollbacks enabling users to restore a previous version quickly. A version includes the main application, dependencies, and all the mechanisms required to deliver the service, such as networking and load balancing.

Armory also supports automated Kubernetes rollbacks in Spinnaker, allowing you to roll back Kubernetes deployments based on complexity and the number of deployed artifacts. A deployment pipeline may fail for many different reasons, but you only need to do a rollback when there is a broken piece of code. This feature lets you configure your settings to guard against unwanted rollbacks by using conditional checks with Spring expressions to identify when you are deploying something broken.

Faulty deployments and inefficient change management are at the top of the list of things that lead to downtime. Downtime can lead to huge losses for organizations to the tune of several millions of dollars and a bad reputation for the technology service provider. As much as we strive to carry out the necessary checks to reduce the frequency of bad deployments, we can’t completely rule out the possibility. Armory helps to improve the quality-of-service delivery in your business through fast-paced and efficient implementations and deployment governance. Armory offers a high level of stability and reliability, ensuring that only quality code gets to the end-user. It does this by using repeatable pipelines to deliver software, enforcing best practices in an organization.

Summary and Conclusion

This article examined what CD is. It also looked at some of the challenges of the CD process, with particular emphasis on rollbacks. The article showed that you could prevent rollbacks altogether by employing the right tools. It drew examples from some features offered by Armory like blue/green deployments that run a production and test server side-by-side and canary deployments where you deploy new releases to a small sub-set of servers first to ascertain their viability. Finally, the article posited that despite due diligence and carrying out all proper checks, it’s still possible for some broken pieces to slip through and get into production, in which case, you must roll back the deployment.

Armory is still there to save the day with its one-click rollback features. Armory’s automated rollbacks enable users to restore a previous version quickly with consistency and dependability. This includes automated rollbacks of Kubernetes containers, which make the process painless for the development team and the end user.

We invite you to check out Armory. Contact Armory today to explore all the exciting features, request a demo, a complimentary assessment of your software delivery practices, or contact support with your inquiries.

The post How to Take the Pain of Rollbacks out of Deployments appeared first on Armory.

How Armory’s Policy Engine can Improve our Nation’s Cybersecurity

stephenatwell — Wed, 19 May 2021 23:38:05 +0000

Earlier this month President Biden issued an executive order on cybersecurity following the Colonial Pipeline Hack. It outlines several actions that government departments must take, and requires them to adopt modern software development and security best practices. This post will explore several of these best practices and explain how Armory’s policy engine addresses them. We will also explore how Armory’s approach differs from other available solutions.

Compliance Checks for Deployed Infrastructure

A major theme in the executive order is ensuring that deployed infrastructure complies with policies. There are three ways companies enforce policies for deployed infrastructure today. Sometimes companies only adopt one of these approaches but ideally, they should utilize a multi-layered approach to security. Leveraging multiple types of controls maximizes the likelihood that even if one control fails, another will still keep your systems safe. An additional layer of security is offered by each of these three approaches, and they are best combined.

Written Policies

The first thing most companies do is to create a formal policy. It will be written down, staff will be trained, and the policy will be audited.

Consider an example policy requiring encryption for data at rest. The auditor selects a random set of data volumes, and checks to see whether or not they are encrypted. While this type of audit is a great backup plan, it is not sufficient to ensure the policy is never violated. In between the audit periods your data may be sitting unencrypted. When the audit does arrive only a subset of data volumes are typically audited, so the unencrypted data may fail to be identified.

While written policies and audits are important, they are inadequate on their own to keep your information secure. However, they serve as a safety net in case other controls break.

Security Scanners

Many companies run security scanning software. This software runs on a schedule (often nightly) and automatically performs an audit. Typically these scanners check specific things, such as volume encryption, or the presence of common vulnerabilities.

Scanners improve on a manual audit process by allowing the audit to happen more frequently, and against all resources instead of just a subset. However, these scanners merely decrease the time to detect a vulnerability, they do not prevent it from being introduced in the first place. Many security scanners also lack automated remediation, further increasing the amount of time during which the vulnerability exists.

Deploy-Time Policy Enforcement

There are a small number of tools (Such as Armory Policy Engine, or Amazon Config Rules) that actively enforce your policy at deploy time. In the case of Armory’s policy engine, the policy will prevent the deployment entirely if it violates policy. In the case of an Amazon Config Rule, a check is run immediately after a configuration change, and an automated remediation action can be triggered.

Policy engine allows your deployment pipeline to check automated rules at deploy time. If any of these rules fail, the deployment will be rejected. With Policy Engine the insecure software never enters production. Of the options discussed this is the only option that keeps the security vulnerability from ever being deployed. This approach also pairs well with your existing security scanners. The security scanners can monitor for new vulnerabilities after deploy (Such as newly identified CVEs), while Policy Engine prevents code with known vulnerabilities from being deployed in the first place. Armory can also provision test environments, and require a pass of the security scanner as part of your deployment pipeline. We’ll talk more about this in the ‘Run Security Scans Before Deployment’ section.

Here are some examples of deployed infrastructure policies required by Biden’s executive order, and how Policy Engine can enforce them.

Section 4.e.i.E: requires employing encryption for data at rest.

In Kubernetes, persistent volume claims can specify what storage class they want to use. Whether or not data in encrypted will typically vary by storage class. For example, here is how you create a new encrypted storage class for Amazon EKS.

The policy below prevents application developers from deploying on any storage class that has not been approved. It also prevents infrastructure engineers from provisioning a storage class that is unencrypted. These two separate layers of enforcement provide a multi-layered approach to ensuring data is always on encrypted volumes. Both controls would need to fail before data could be deployed on an unencrypted volume. This policy expects the list of approved storage classes to be loaded into OPA’s data document, in an array named ‘approvedStorageClasses’:

package spinnaker.execution.stages.before.deployManifest

deny [“You cannot create unencrypted storage classes”] {

some i

input.stage.context.manifests[i].kind==”StorageClass”

params := object.get(input.stage.context.manifests[i],”parameters”,null)

object.get(params,”encrypted”,”false”)!=”true”

}

deny [“You must use an approved encrypted storage class”]{

some i

input.stage.context.manifests[i].kind=”PersistentVolumeClaim”

storageClass := object.get(input.stage.context.manifests[i].spec,”storageClassName”,null)

not isApprovedStorageClass(storageClass)

}

isApprovedStorageClass(storageClass){

(storageClass == data.approvedStorageClasses[_])

}

Section 3.d: requires encryption of data in transit.

Whether or not data is encrypted in transit depends on what network protocols are used. For example, HTTP is not encrypted, but HTTPS offers the same functionality in an encrypted form. Different protocols use different network ports, and Policy Engine can enforce policies that regulate what network ports can be exposed. For example, here is a policy that prevents exposing the following protocols while allowing their more secure equivalents: HTTP, FTP, TELNET, POP3, NNTP, IMAP, LDAP, SMTP. This example prevents both pods and services from being deployed that expose these ports:

package spinnaker.deployment.tasks.before.deployManifest

blockedPorts :=[20,21,23,80,110,119,143,389,587,8080,8088,8888]

deny[“You cannot leverage unencrypted network protocols. A port typically used by an unencrypted protocol was detected.”] {

check if it is a service with either an insecure port or target port.

manifest := input.deploy.manifests[_]

manifest.kind == “Service”

port := manifest.spec.ports[_]

any([object.get(port,”port”,null) == blockedPorts[_],

object.get(port,”targetPort”,null) == blockedPorts[_]])

} {

check if it is a pod exposing an insecure port.

input.deploy.manifests[_].spec.containers[_].ports[_].containerPort=blockedPorts[_]

} {

Check if it contains a pod template exposing an insecure port.

input.deploy.manifests[_].spec.template.spec.containers[_].ports[_].containerPort=blockedPorts[_]

}

Run Security Scans Before Deployment

President Biden’s Executive Order requires “employing automated tools, or comparable processes, that check for known and potential vulnerabilities and remediates them…at a minimum prior to product, version, or update release.” Many companies execute security scanners against deployed code prior to releasing it, but this is often manual, and rarely done with enough control to ensure it happens every time code ships. Because it is a manual process companies sometimes only enforce the scan for a subset of releases, which creates a security hole.

Analyze Pipeline Structure

Armory allows your application delivery teams to automate your software delivery pipeline. While doing so, Policy Engine can analyze the pipeline steps to ensure that necessary process checks take place. This allow you to, for example, write a policy that prohibits deploying to a production account unless your security scans have already been run in a staging environment. Here is an example policy that simply requires a manual approval prior to any deployment to a production account:

package opa.pipelines

production_accounts := [“prod1″,”prod2”]

deny[“production deploy stage must follow a manual judgement stage”] {

some j

input.pipeline.stages[j].type==”deployManifest”

input.pipeline.stages[j].account==production_accounts[_]

approvers := [i | input.pipeline.stages[i].type==”manualJudgment”; i<j]

count(approvers)==0

}

Role based access control can also be leveraged to restrict who can provide the approval. This ensures that the personnel responsible for each control have signed off before deployment.

Alternately the manual approval stage can be replaced with any other pipeline stage, for example a stage that automates the security check. Some of our customers require all pipelines to have security scanning stages prior to any production deployment stage. In this way they require application developers to deploy their application to staging, and run a security scanner before deploying it to production. However, for the application developers it is a single automated deployment pipeline that they can control. The pipeline deploys to staging, scans, and finally deploys to production if everything looks good.

Integrate with External Systems

If your security scanners provides a list of scanned and approved images, Policy Engine can also be configured to prohibit deployment of any images that are not on the list. Policy Engine Policies are executed by Open Policy Agent (OPA). OPA allows external systems to load such data into its data document, and also allows policies to invoke APIs to retrieve such data. This example assumes that you have loaded an array named ‘approvedImages’ into the OPA data document, and it will reject manifests that leverage images that are not in the approved list:

package spinnaker.deployment.tasks.before.deployManifest

deny[“Your manifest creates a pod from an image that has not been approved by the security scanning process.”] {

Check pod templates

isImageUnApproved(input.deploy.manifests[_].spec.template.spec.containers[_].image)

} {

check pods

isImageUnApproved(input.deploy.manifests[_].spec.containers[_].image)

}

{

check pod template initContainers

isImageUnApproved(input.deploy.manifests[_].spec.template.spec.initContainers[_].image)

}

{

check pod initContainers

isImageUnApproved(input.deploy.manifests[_].spec.initContainers[_].image)

}

isImageUnApproved(image){

not isImageApproved(image)

}

isImageApproved(image){

image==data.approvedImages[_]

}

Reporting Requirements

While details about the reporting requirements are not in the executive order, the executive order makes it clear that there will be reporting requirements for national security systems. Agencies will also need to establish log management practices that provide visibility to the highest level security operations center in each agency.

Armory policy engine allows you to easily assemble a history of all changes to your production environment. Anytime Armory deploys code, it will perform a policy check. The policy check logs details of what is being deployed, who triggered the deploy, as well as the results of evaluating the policy. This information is logged in Open Policy Agent’s Decision logs which can be collected in a log analytics tool to enable easy auditing of historic changes to production.

Wrapping Up

This executive order requires improvement in all government agencies and contractors. It acknowledges that the security of our nation is only as good as the security of our software supply chain.

However, this should be a wake up call to corporate America as well. Colonial Pipeline was a private company, not a government agency. This hack massively disrupted their business, and our country. We all have a duty to maintain the security of our critical systems, and for software this starts by securing our software delivery pipelines. If you are not confident in the security of your software delivery pipeline, request a demo of how Armory can improve your security by automating your policy as code.

The post How Armory’s Policy Engine can Improve our Nation’s Cybersecurity appeared first on Armory.