DEV Community: Ihechi Okere

Troubleshooting Bicep Updates on Windows Self-Hosted Agents

Ihechi Okere — Fri, 01 Aug 2025 00:26:54 +0000

🛠️ Resolving Bicep Update Issues in Self-Hosted CI/CD Agents

This post is a follow-up to my previous article on updating Bicep to leverage Microsoft's latest features. In our organization, we use Windows-based self-hosted agents for CI/CD pipelines. While updating Bicep might seem as simple as running a command and moving on, I recently encountered a subtle issue that proved otherwise.

⚠️ The Problem

After running the update command for Bicep, I expected the pipeline to use the latest version. However, it continued to execute using an older version. This was puzzling, especially since the update process had completed without errors.

Initial Troubleshooting Approach

Initially, I suspected a path conflict. Months ago, I had manually placed the Bicep executable in Program Files to avoid copying it over with each update. To streamline future upgrades, I configured a system environment variable pointing to the auto-updated version located in:

C:\Users\<username>\.azure\bin\bicep.exe

Despite confirming that the system path was correctly set, the pipeline still defaulted to the outdated executable.

🔍 Root Cause Discovery

To investigate further, I added a whoami command to the PowerShell script running in the pipeline:

whoami

Surprisingly, the job was being executed by:

NT AUTHORITY\NETWORK SERVICE

This was not the account I had assumed.

✅ The Solution

This discovery led me to check the file permissions on the Bicep executable. Since the machine was domain-joined but the service account was local, I had to explicitly grant NETWORK SERVICE the necessary permissions to read and execute the file.

Here's what I did:

Right-clicked the bicep.exe file
Opened Properties > Security
Added NETWORK SERVICE to the list of users
Granted Read & Execute permissions

Once I updated the file's security settings, the pipeline successfully picked up the correct version of Bicep.

💡 Key Takeaways

If you're using Windows self-hosted agents and encounter issues with updating Bicep—or any other tool—it's worth checking:

Service Account Context: Always verify which account is actually executing your pipeline processes
Permission Management: System-level path configurations are insufficient if the executing service account lacks proper file permissions
Local vs. Domain Accounts: Consider the account type when troubleshooting permission-related issues

This solution applies broadly to similar scenarios involving executable updates for any programs within Windows self-hosted agents, not just Bicep.

Hopefully, this helps someone avoid the same rabbit hole I went down! If you've run into similar issues or have other tips for managing self-hosted agents, feel free to share in the comments.

Thanks for reading! 👋

Secure Outputs in Bicep

Ihechi Okere — Sat, 05 Jul 2025 00:24:26 +0000

🔐 Securing Bicep Outputs: A Game-Changer for Logic App Deployments

While developing a module to deploy a Logic App using Azure Bicep, I hit a frustrating snag: I couldn’t securely hide one of the outputs.. specifically, the storage account keys. The issue arose when I needed to call the storage account module from within the Logic App module. If a certain parameter was set to true, it would create a new storage account. Simple enough, right?

Well, not quite.

The Problem: Exposed Secrets

Every time I tried this setup, I ran into a Bicep linting error. More importantly, I was uneasy about the fact that the storage account key would always be exposed in the deployment output. Anyone with access to the deployment history could easily view it under:

Resource Group → Settings → Deployments

Here’s what that looks like:

This meant that even sensitive values like storage keys were visible in plain text—an obvious security risk, especially in shared environments or production scenarios.

My Temporary Solution: Separation of Concerns

At the time, the only workaround was to separate the storage account module from the Logic App deployment entirely. It wasn’t elegant, but it avoided exposing secrets. I shelved the problem, hoping for a better solution down the line.

The Comeback: Enter `@secure()` for Outputs

And then—plot twist! Microsoft dropped a powerful new feature in Bicep v0.35.1: the @secure() decorator for outputs. 🎉

Previously, @secure() was only available for input parameters. Outputs, no matter how sensitive, were always exposed in plain text. But now, you can mark outputs as secure, and they’ll be hidden from the deployment logs.

This is a huge win for anyone managing secrets or sensitive infrastructure details in Bicep.

Example Usage

@secure()
output storageKey string = storageAccount.listKeys().keys[0].value

With this, the output will no longer appear in plain text in the deployment history. Instead, it’s masked—just like secure parameters.

Still Seeing Linting Errors?

If you’re still getting linting errors when using @secure() on outputs, chances are you’re running an older version of Bicep.

🔍 Check your version:

az bicep version

📦 Install a specific version:

az bicep install --version v0.35.1

⬆️ Or upgrade to the latest:

az bicep upgrade

Learn More:

For full details, check out the official Microsoft documentation:

Secure Outputs in Bicep

This small but mighty feature makes Bicep even more production-ready. If you’ve ever hesitated to use outputs for sensitive data, now’s the time to revisit your templates.

Have you tried the new @secure() decorator yet? I’d love to hear how it’s changed your workflow or if you’ve found other clever ways to manage secrets in Bicep.

A Safer Container Runtime

Ihechi Okere — Tue, 01 Apr 2025 00:24:03 +0000

Security experts within the container ecosystem have been sounding the alarm lately: Kubernetes, the darling of container orchestration, is likely to become the next major frontier for cyberattacks. As one prominent DevSecOps engineer recently put it, "We're sitting on a security time bomb with Kubernetes deployments. The complexity that makes K8s powerful also creates an expanding attack surface that most teams aren't prepared to defend."

I couldn't agree more with this assessment. With the use of containerization technology not slowing down anytime soon, it's only a matter of time before threat actors focus their efforts on containers and container orchestration platforms and low hanging fruits will be the first to be grabbed.

Rootless containers provide a way to run containers safely by running the container with a user other than the root user or running the container in privileged mode. By default, docker containers run with root privileges. This is because the Docker daemon is required to run containers and it has to run with root privileges in order to provide some powerful Docker features. You can run a container where the /host directory is mounted on the / directory of the host without restrictions. Such is the power afforded to users of the Docker daemon. However, this adds an added risk of compromise of the host which runs the daemon. But by leveraging rootless containers, your applications can be run in containers that are created, run, and managed by users without admin rights. It also adds a new layer of security by utilising identity management to prevent attackers from gaining root privileges on the host even if the container engine, runtime, or orchestrator is compromised.

Below are ways we can safely run containers to minimise the risks associated with running containers as root or in privileged mode:

Set containers root filesystem as read-only
Run as a non-root user
Build rootless containers using supported tools

Running containers do not need changes to be made to the root filesystem. Attempted changes to the root filesystem often is an indication of compromise and is mostly attempted by malicious actors. The immutable nature of containers should be preserved.

Building Rootless Containers

To build truly secure containers, you need to start with the build process itself. Traditional Docker builds run as root by default, but there are better alternatives:

Kaniko

Kaniko is a tool from Google that builds container images from a Dockerfile inside a container or Kubernetes cluster without requiring privileged mode:

# Example running kaniko in a Kubernetes pod
kubectl run kaniko \
  --image=gcr.io/kaniko-project/executor:latest \
  --restart=Never \
  --env="DOCKER_CONFIG=/kaniko/.docker" \
  --env="GOOGLE_APPLICATION_CREDENTIALS=/secret/kaniko-secret.json" \
  --volumes=kaniko-secret:/secret \
  -- \
  --dockerfile=Dockerfile \
  --context=gs://my-bucket/my-app \
  --destination=my-registry/my-app:latest

Kaniko executes each command within the Dockerfile completely in userspace, which makes it a safer alternative to Docker's build process.

Buildah

Buildah is another tool that can build OCI container images without requiring root privileges:

# Install buildah
sudo apt-get install buildah

# Run buildah in rootless mode
buildah --userns=keep-id build-using-dockerfile -t my-app:latest .

# Push to a registry
buildah push my-app:latest docker://registry.example.com/my-app:latest

Buildah can be configured to run in a fully rootless mode, making it an excellent choice for CI/CD pipelines where security is a priority.

When combined with tools like Podman for running containers, you can achieve a fully rootless container workflow from build to execution.

Confidential Containers

Confidential containers represent an emerging approach to container security. They utilize hardware-based Trusted Execution Environments (TEEs) to protect sensitive data and workloads even from privileged users on the host system. This technology creates a secure enclave where container workloads can run with enhanced privacy and integrity guarantees.

Drop Capabilities

Another important security measure is dropping unnecessary Linux capabilities. Docker containers, by default, run with a reduced set of capabilities compared to a normal root user, but you can further restrict these:

docker run --cap-drop=ALL --cap-add=NET_BIND_SERVICE your-image

This removes all capabilities and only adds back the specific ones your application needs, following the principle of least privilege.

Security Scanning

Implement automated container image scanning in your CI/CD pipeline. Tools like Grype, Clair, and Anchore can detect vulnerabilities in your container images before they're deployed to production:

grype image your-image:latest

Use Container-Specific Operating Systems

Minimal, purpose-built operating systems like Alpine Linux, Bottlerocket, or Google's Container-Optimized OS reduce the attack surface by eliminating unnecessary packages and services.

Secure Your Container Runtime

Keep your container runtime up-to-date and properly configured. For example, if using containerd:

# Check for security-related configurations
containerd config default | grep -i security

Network Policies

Implement strict network policies to control the traffic between containers:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny
spec:
  podSelector: {}
  policyTypes:
  - Ingress
  - Egress

This example blocks all incoming and outgoing traffic for pods by default.

Runtime Security Monitoring

Deploy runtime security tools that can detect and respond to suspicious activities inside containers. Solutions like Falco can provide real-time alerts on container behavior:

- rule: Terminal shell in container
  desc: A shell was used as the entrypoint/exec point into a container
  condition: container and shell_procs and not container_entrypoint
  output: Shell running in a container (user=%user.name container_id=%container.id container_name=%container.name shell=%proc.name parent=%proc.pname cmdline=%proc.cmdline)
  priority: WARNING

OCI Runtime Security Profiles

Leverage OCI runtime security profiles like seccomp and AppArmor to restrict what system calls and operations containers can perform:

docker run --security-opt seccomp=/path/to/seccomp/profile.json your-image

Conclusion

Container security requires a multi-layered approach. By implementing rootless containers, read-only filesystems, runtime security controls, and proper monitoring, you can significantly reduce the risk of container-based attacks. As containerization continues to grow in popularity, investing in these security measures now will help prevent your organization from becoming an easy target for attackers looking to exploit container vulnerabilities.

It is worth remembering that container security should be a continuous process, not just a one-time setup. Regular audits, updates, and staying informed about the latest security best practices and vulnerabilities are essential parts of maintaining a secure container environment.