DEV Community: Jakub Wołynko

The standard - S3 IAM Policies

Jakub Wołynko — Tue, 24 Jun 2025 07:42:45 +0000

In the previous post, we explored S3 Access Control Lists (ACLs)
and learned why AWS recommends disabling them for most modern use cases.
Now it's time to dive into the proper way of securing your S3 buckets:
IAM policies and bucket policies.

Unlike ACLs, which are considered legacy and can become operationally chaotic,
IAM policies offer centralized, scalable, and auditable access management.
They're the foundation of modern AWS security architecture and the tool
you should reach for when securing your S3 resources.

Today we'll focus on IAM policies - the backbone of AWS access control that actually makes sense.

IAM Policies vs Bucket Policies - What's the Difference?

Before we dive deep, let's clarify the two main policy types for S3:

IAM Policies are attached to IAM users, groups, or roles and define what actions
those identities can perform across AWS services. They're identity-centric -
"what can this user/role do?"

Bucket Policies are attached directly to S3 buckets and define who can
access that specific bucket and what they can do with it. They're resource-centric -
"who can access this bucket?"

Both use the same JSON policy language, but they serve different purposes
and are evaluated together by AWS when determining access. Think of them as
complementary layers of security rather than competing mechanisms.

Understanding IAM Policy Structure

IAM policies follow a standardized JSON structure that's both powerful and
relatively straightforward once you understand the components:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowS3ReadAccess",
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:ListBucket"
      ],
      "Resource": [
        "arn:aws:s3:::my-secure-bucket",
        "arn:aws:s3:::my-secure-bucket/*"
      ],
      "Condition": {
        "IpAddress": {
          "aws:SourceIp": "203.0.113.0/24"
        }
      }
    }
  ]
}

Let's break this down:

Version: Currently, '2012-10-17' is the recommended and most up-to-date policy language version. It's best practice to use this version for all new policies.
Statement: An array of individual permission statements
Sid: Optional statement identifier for easier management
Effect: Either "Allow" or "Deny"
Action: The specific AWS API actions being granted/denied
Resource: The AWS resources the policy applies to
Condition: Optional conditions that must be met for the policy to apply

Essential S3 Actions and Resource Best Practices

Understanding the right S3 actions is crucial for creating effective policies.
Here are the most commonly used ones:

Bucket-Level Actions

{
  "Action": [
    "s3:ListBucket",           // List objects in bucket
    "s3:GetBucketLocation",    // Get bucket region
    "s3:GetBucketVersioning",  // Check versioning status
    "s3:ListBucketVersions"    // List object versions
  ],
  "Resource": "arn:aws:s3:::my-bucket"
}

Object-Level Actions

{
  "Action": [
    "s3:GetObject",          // Download objects
    "s3:PutObject",          // Upload objects
    "s3:DeleteObject",       // Delete objects
    "s3:GetObjectVersion",   // Get specific object versions
    "s3:DeleteObjectVersion" // Delete specific versions
  ],
  "Resource": "arn:aws:s3:::my-bucket/*"
}

Notice how bucket-level actions use the bucket ARN (arn:aws:s3:::my-bucket)
while object-level actions use the object ARN pattern (arn:aws:s3:::my-bucket/*).

Practical IAM Policy Examples

Read-Only Access Policy

Perfect for analytics tools or monitoring systems that need to read data
but shouldn't modify anything:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:ListBucket"
      ],
      "Resource": [
        "${aws_s3_bucket.analytics_data.arn}",
        "${aws_s3_bucket.analytics_data.arn}/*"
      ]
    }
  ]
}

In Terraform, always use resource references:

# Good - uses Terraform resource reference
resource "aws_iam_policy" "analytics_read" {
  name = "AnalyticsReadAccess"

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect = "Allow"
        Action = [
          "s3:GetObject",
          "s3:ListBucket"
        ]
        Resource = [
           aws_s3_bucket.analytics_data.arn,
          "${aws_s3_bucket.analytics_data.arn}/*"
        ]
      }
    ]
  })
}

# Bad - hardcoded bucket name
resource "aws_iam_policy" "analytics_read_bad" {
  policy = jsonencode({
    Resource = [
      "arn:aws:s3:::company-analytics-data",  # Don't do this!
      "arn:aws:s3:::company-analytics-data/*"
    ]
  })
}

Application Upload Policy

For applications that need to upload files but shouldn't be able to
delete or list existing content:

# Define the policy using Terraform data source
data "aws_iam_policy_document" "app_upload" {
  statement {
    effect = "Allow"
    actions = [
      "s3:PutObject",
      "s3:PutObjectAcl"
    ]
    resources = ["${aws_s3_bucket.app_uploads.arn}/*"]

    condition {
      test     = "StringEquals"
      variable = "s3:x-amz-server-side-encryption"
      values   = ["AES256"]
    }
  }
}

resource "aws_iam_policy" "app_upload" {
  name   = "AppUploadPolicy"
  policy = data.aws_iam_policy_document.app_upload.json
}

This approach ensures your policy always references the correct bucket,
regardless of environment or bucket naming changes.

Time-Based Access Policy

Sometimes you need to restrict access to specific time windows:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
      ],
      "Resource": [
        "arn:aws:s3:::business-hours-data/*"
      ],
      "Condition": {
        "DateGreaterThan": {
          "aws:CurrentTime": "08:00:00Z"
        },
        "DateLessThan": {
          "aws:CurrentTime": "18:00:00Z"
        }
      }
    }
  ]

Common Pitfalls and Security Considerations

The Wildcard Trap

One of the most dangerous mistakes is using overly broad wildcards:

// DON'T DO THIS
{
  "Effect": "Allow",
  "Action": "s3:*",
  "Resource": "*"
}

This grants unlimited S3 access across your entire AWS account.
Always be specific about actions and resources.

Resource ARN Best Practices

Avoid hardcoded bucket names in your policies! Using fixed bucket names like
arn:aws:s3:::my-bucket creates several problems:

Policies become environment-specific and hard to reuse
Risk of accidentally granting access to wrong buckets
Makes bucket renaming nearly impossible
Creates maintenance nightmares across multiple environments

Instead, use variables and references.

As it's overview blog-post, sometime I used harcoded buckets name, in sake of simplicity.

Missing Bucket vs Object Permissions

A common source of confusion is forgetting that listing bucket contents
and reading objects require different permissions on different resources:

// Correct approach
{
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "s3:ListBucket",
      "Resource": "arn:aws:s3:::my-bucket"  // Note: no /*
    },
    {
      "Effect": "Allow", 
      "Action": "s3:GetObject",
      "Resource": "arn:aws:s3:::my-bucket/*"  // Note: with /*
    }
  ]
}

Cross-Account Access Considerations

When granting cross-account access, always use explicit conditions
to prevent unauthorized access:

{
  "Effect": "Allow",
  "Action": "s3:GetObject",
  "Resource": "arn:aws:s3:::shared-bucket/*",
  "Condition": {
    "StringEquals": {
      "aws:SourceAccount": ["123456789012", "987654321098"]
    }
  }
}

Implementing S3 IAM Policies with Terraform

Terraform makes managing IAM policies much more maintainable than
clicking through the AWS console. Here's how to implement the policies
we discussed:

Basic Policy Attachment

# Create the IAM policy
resource "aws_iam_policy" "s3_read_only" {
  name        = "S3ReadOnlyAccess"
  description = "Read-only access to specific S3 bucket"

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect = "Allow"
        Action = [
          "s3:GetObject",
          "s3:ListBucket"
        ]
        Resource = [
          aws_s3_bucket.app_data.arn,
          "${aws_s3_bucket.app_data.arn}/*"
        ]
      }
    ]
  })
}

# Attach to a role
resource "aws_iam_role_policy_attachment" "s3_read_only" {
  role       = aws_iam_role.app_role.name
  policy_arn = aws_iam_policy.s3_read_only.arn
}

Using Data Sources for Flexibility

For more complex scenarios, you can use Terraform data sources
to make your policies more dynamic:

data "aws_iam_policy_document" "s3_upload_policy" {
  statement {
    effect = "Allow"

    actions = [
      "s3:PutObject",
      "s3:PutObjectAcl"
    ]

    resources = [
      "${aws_s3_bucket.uploads.arn}/uploads/${var.environment}/*"
    ]

    condition {
      test     = "StringEquals"
      variable = "s3:x-amz-server-side-encryption"
      values   = ["AES256"]
    }

    condition {
      test     = "IpAddress"  
      variable = "aws:SourceIp"
      values   = var.allowed_ip_ranges
    }
  }
}

resource "aws_iam_policy" "s3_upload" {
  name   = "S3UploadPolicy-${var.environment}"
  policy = data.aws_iam_policy_document.s3_upload_policy.json
}

Policy Validation and Testing

Always validate your policies before applying them. Terraform can help
catch syntax errors, but logic errors require testing:

# Use locals for policy validation
locals {
  # Validate bucket ARN format
  bucket_arn_pattern = "^arn:aws:s3:::[a-z0-9][a-z0-9\\-]*[a-z0-9]$"

  # Ensure bucket name follows naming conventions
  valid_bucket_name = can(regex("^[a-z0-9][a-z0-9\\-]*[a-z0-9]$", var.bucket_name))
}

# Use validation blocks
variable "bucket_name" {
  type        = string
  description = "S3 bucket name"

  validation {
    condition     = can(regex("^[a-z0-9][a-z0-9\\-]*[a-z0-9]$", var.bucket_name))
    error_message = "Bucket name must contain only lowercase letters, numbers, and hyphens."
  }
}

Policy Testing and Validation

The AWS IAM Policy Simulator is your best friend for testing policies
before deployment. It allows you to simulate API calls and see whether
they would be allowed or denied by your policies.

For Terraform users, consider using the aws_iam_policy_document data source
instead of hardcoded JSON - it provides better syntax validation and
makes policies more readable.

Summary

IAM policies are the modern, scalable way to secure S3 resources.
Unlike ACLs, they provide centralized management, powerful conditions,
and excellent auditability. Key takeaways:

Use specific actions and resources - avoid wildcards
Remember bucket-level vs object-level permissions require different resource ARNs
Leverage conditions for additional security controls
Test policies thoroughly before production deployment
Use Terraform for maintainable, version-controlled policy management

The combination of IAM policies and bucket policies gives you fine-grained
control over who can access your S3 resources and what they can do with them.
In the next article, we'll explore bucket policies and how they complement
IAM policies to create a comprehensive S3 security strategy.

As always, even with a two-month-old keeping me busy, I'm committed to
sharing practical AWS security knowledge. The NixOS experiment is currently
stopped, however I starting thinking about re-activation of my newsletter.
Who knows, for sure the format will be diffrent, as link aggregation is just
boring!

Useful links

Fighting Bots While Serving Humans: A Real-World API Protection Journey

Jakub Wołynko — Mon, 16 Jun 2025 10:11:16 +0000

In today's digital landscape, serving APIs has become increasingly challenging. The task of exposing data and functionality to legitimate users has evolved into a complex battle against automated attacks, aggressive crawlers, and misguided AI training attempts.

The challenge isn't just technical—it's philosophical.

How do you distinguish between legitimate automation (which powers the modern web) and malicious bots that drain resources and compromise security?

How do you balance accessibility with protection, especially when dealing with rate limiting, bot detection, and content delivery networks like Cloudflare?

This post explores the real-world challenges of maintaining APIs that serve humans while fighting with army of automated requests that seem determined to treat every endpoint as their personal data source.

Intro

As I enjoy working with multiple folks, one of my colleagues is running a company that collects a lot of data. Really cool data. The goal is to give everyone on the Internet a chance to see it or work with them. The problem on the other hand is that is not a real-time system, data are updated once a day. So hitting API every second is emm, pointless as you get exactly the same date 86400 times.

And it's a problem on my side, as it just uses a lot of resources, especially if we have multiple IPs that work in that way. So what's the scale of the challenge? For less than ~ 40K requests in 24 hours, mostly on Sunday.

Iteration 1

Ah, it's time to clarify, we're using Cloudflare at the front, and then API is hosted in different places, this particular one is using Azure App Services. (yep, I can into Azure as well). But that's not a point, I'm not a developer, I was contracted here to just do a security/network job.

My first idea was to observe data from the Events tab and try to understand the pattern. For me blocking a particular IP wasn't an option.

Why? This requires a bit of specific knowledge. As you may know,
I'm living in Poland, and having a static IP it's not a default option. If you need it, you need to pay a bit more extra for the business plan in most cases. Based on that fact, and information that our. My favourite client is from Poland as well (at least based on CloudFlares' data), blocking just IP doesn't have sense,
as it will change after some time, and we will block random person
or organization on the internet.

The first iteration was the usage of the classic rate limit. Also, note, that it's partly supported by Terrafrom modules, so our code was more or less:

resource "cloudflare_ruleset" "rl_custom_response" {
  zone_id     = "<ZONE_ID>"
  name        = "Rate-limit bots"
  description = ""
  kind        = "zone"
  phase       = "http_ratelimit"

  rules {
    ref         = "rate_limit_example_com_status_404"
    description = "Rate limit requests to www.example.com when exceeding the threshold of 404 responses on /status/"
    expression  = "http.host eq \"www.example.com\" and (http.request.uri.path matches \"^/status/\")"
    action      = "block"
    action_parameters {
      response {
        status_code  = 429
        content      = "{\"response\": \"API refresh data, one a day. Come back tommorow\"}"
        content_type = "application/json"
      }
    }
    ratelimit {
      characteristics     = ["ip.src", "cf.colo.id"]
      period              = 10
      requests_per_period = 5
      mitigation_timeout  = 30
      counting_expression = "(http.host eq \"www.example.com\") and (http.request.uri.path matches \"^/status/\") and (http.response.code eq 404)"
    }
  }
}

There were no changes after that, we got a lot of blocked IPs, but they are still pinging the API. That was great evidence, that bots are used for home automation or some not very important system.
Also, it introduces a problem, we have no idea how users use the data. After a while, we realised, that a lot of browser-based users were blocked. Probably due to the constant refreshing of some kinds of dashboards.

In the meantime, we changed the custom response, to Managed Challange. Unfortunately, it doesn't solve the issue with user behaviours.

Iteration 2

At this point, we're sure, that the rate limit without knowledge about API usage could be a challenge. So we've been forced to find a different solution.

The next step was to block IPs to at least cut off, the most active
autmated users. As it works, as expected, we have noticed a lot of requests classified by Cloudflare as Azure/AWS/etc. What it could mean? Probably some Machine learning processes. Again, blocking whole hyperscaler subnets is just pointless, we can't block the whole Internet, right?

Iteration 3

At this point after tracking an events dashboard, I realized
that there is a simple way to block "not-browser" users. Check a user agent. After a short debug session, I was able to build the first user-agent-based block query. Which looks like that:

(http.user_agent eq "") or
(http.user_agent eq "got (https://github.com/sindresorhus/got)") or
(http.user_agent eq "Embarcadero RESTClient/1.0") or
(starts_with(http.user_agent, "python-requests"))

This very simple method allows me to drop all, non-human requests.
So far so good, right?

Iteration 4

After a while and a few days of watching events, I realized that
a few requests are made by AI crawlers.

That's led to more philosophical questions. Should we treat it as users who are seeking knowledge, and GenerativeAI finds out our data is valuable. Or we should block it, as it does not generate traffic to our blogs, or app, just use the data without linking to it? Also, it will probably include results and path in the newly trained model, what will be the situation when path is updated, and the model will redirect the user to the wrong source? Who will be responsible for it? Our company, or model owner? To just avoid these questions, we will block it for now. If you have a different option let me know!

Summary

This not-such-long article could be summarised in a few lines of text, and it is exactly what I will do.

To protect your API, no matter what tool you use, you need a few steps:

Observe events and logs of your API
Try to find a pattern
Implement a rather, simple, not aggressive solution
Keep observing, try to understand users, not developers
Implement another rule, based on the results
Keep integrating until, your resources are happy, as well as your clients, and wallet (if you're using the cloud)

That's for reading it, if you find it interesting, or you have different observation ping me on Mastodon or via email.

How do I use Vercel to host Umami?

Jakub Wołynko — Tue, 15 Apr 2025 07:42:31 +0000

As you may already know, or not. I'm self-hosting a view apps, where it doesn't mean home-labbing them. My flat wasn't designed to have network cables in all rooms, and storing more units than Synology NAS, HomeAssistant, RasbberyPi and Router on a tiny Ikea shelf could be a dangerous situation. That is why I'm not using my mini-pc as a homelab anymore.

In the beginning, I moved a lot of my services to Hetzner (take a look at previous posts). Now after a while, I started thinking about making more for less. During the vibe coding wave, I learned how to use Supabase and Vercel. Then I started thinking about one crazy thing.

Can I host Umami on Vercel and Supabase for free?

At the end of March, some free time appears. That is why you my friend can read this blog post!

What is Umami, Vercel or Supabase?

Let's rely on the provided documentation/marketing slogans.

Umami

Umami makes it easy to collect, analyze,
and understand your website data — so you can focus on growth

Basically Umami is open-source tool that allows us - content creators collect data about our users. For example, I like to track the way users come to my blog or most popular articles, so I can modify pinned ones. Also, it's focused on privacy. For example, user data are analyzed by default, also there is no cross-site tracking, so umami tracks users only on a particular website.

Vercel

Vercel provides the developer tools and
cloud infrastructure to build, scale,
and secure a faster, more personalized web.

Vercel is a full CI/CD platform, that provides infrastructure and whole more for your projects. It's expensive when you're bigger, but on my scale the price is great - 0$! Also, they are responsible for next.js, so we can consider them as solid brands. To be totally honest I'm really impressed by their CI/CD system, it works really good for standard apps!

Supabase

Supabase is an open source Firebase alternative.
Start your project with a Postgres database,
Authentication, instant APIs, Edge Functions,
Realtime subscriptions, Storage, and Vector embeddings.

Yes, the Supabase description could be not easy to understand. It's become much more clear when you start using it. Let's say it that way, as Vercel is for hosting the frontend, Supabase will handle your backend. Small Postgresql database, edge functions, also auth to your app, s3-like storage, etc.

Example

For example, you can take a look at my [security scanner (https://www.selfsiteaudit.com/). It was built entirely with the usage of Vercel as web app hosting, Supabase as database(for report storing), and place where my Deno's functions are living - serverless part!

Let's start then

Start with Vercel, the only thing is that we need to open an account there.

As you can see, they do not provide Google/Facebook auth options, fortunately we can just use the email/login method.
Next we need to create our first project.

To achieve that, you need first click Add New on your project dashboard.
After that we need to configure a connection with GitHub, or another VCS server.

But hey! We don't have any projects yet. Let's fix it now.
Go to the GitHub page and fork the offical repository.

Great, now we're almost done.

Let's see Vercel's magic in action.
Back to the project page, and choose your fork.
Just click deploy.

A. You can change the name of the project (but why?)
B. Auto-detected framework.
C. Some boring settings.
D. Click the Deploy button, just do it!
And... bum! Errors.

At this stage, we need a database. It's not that simple right? Vercel is not such smart yet. In the beginning, I thought about Supabase, however I noticed that Vercel has something called Storage.

Read and write directly to databases and stores from your projects

Let's use it then for our application.
Create a database.
Choose a database type.

Here we can choose multiple providers, Supabase as well. Just out of curiosity, I decided to Neon - a serverless version of Postgresql. Just to check if it will work as expected. You can choose whatever you want. After that find your region, as well as Installation Plans, where the recommended version is ehm, Free.
Connect our database.
1. Go to Storage tab.
2. Click Connect Project.
3. Search for your project and choose it.
4. Click Connect.
5. Now you should see something like this:
Now, your app should be fully connected.
Redeploy the app.
1. Overview -> Project name pane -> Deployments.
2. Choose the first build and hit Redeploy
After a while you should see a similar screen to the below one:
1. Here you see the deployment status - Ready
2. As well as the page URL, in my case https://umami-6n48.vercel.app/, just hit it!

Summary

As you can see, you can access your instance of Umami. Initial login, will require some credentials, remember to change them!

username: admin
password: umami

What's the result of this project? Up and running self-hosted an instance of Umami, for free, able to handle analytics at least for 5 different pages, I'm currently running. Why it could be a good move? The ability to learn new things is always good. As it's a self-managed app, you don't need to worry about the price change, also Vercel stability for sure will be better than a regular production-like environment, where you still testing a lot of things. The most important thing is that you don't need to use Google Analytics if you want to use website analytics for free and avoid services provided by Google.

Bootstrap cluster with FluxCD

Jakub Wołynko — Tue, 15 Apr 2025 07:28:36 +0000

That is the next iteration of the "5 AM Club" Kubernetes migration. As you can remember from other entries published in this series, I started playing with Kubernetes daily. Not on a daily basis, but literally every single day. To be honest, I'm pretty happy with the results. However, my plan has one challenge, that requires to be solved. Build time was long ~15 minutes every day, and sometimes ESO operator faced an issue regarding using kustomize for Helm-based deployment. And just due to random constraints, I wanted to use one tool for bootstrapping a cluster.

That is why I started with the layout presented last time. So logical split per operator with the base/overlay sub-group.

Then I thought, why not just use some standard solution for that?
The first idea was to extend the usage of Argo for my infrastructure. But... It requires setting ESO, Doppler secret, and ArgoCD installed. Then I can reconfigure apps from the git level.

The challenge was, to make it easier, faster, and more standardised, than it was currently.

Starting from beginning

What Flux is? And what is not?

Flux is a tool for keeping Kubernetes
clusters in sync with sources of configuration (like Git repositories),
and automating updates to the configuration when there is new code to deploy.

We can use Flux to keep the configuration of our whole cluster in sync, with the Git repository. The funny thing is that we can configure both apps and infra with the usage of Flux.

However, managing apps with Flux in my opinion is not the easiest and most comfortable solution. Especially, if we're changing versions quite often and we would like to have at least a few dependencies between apps and infra. For example, my Immich instance needs csi-driver-smb, which on the other hand requires external-secret-operator and external-secret-secret
(the actual link between in-cluster secret and ESO ClusterSecretStore). So, every new relese, needs to be built and checks that all kustomizations are in place.
Then actually deploy a new version. Very long process, also ArgoCD UI is just better, easier to use, and definitely more user-friendly - at least in my opinion.

Repository structure

So after a few initial rounds it's ended in the following state:

.
├── README.md
├── clusters
│   └── cluster0
│       └── flux-system
│           ├── gotk-components.yaml
│           ├── gotk-sync.yaml
│           ├── infrastructure.yaml
│           └── kustomization.yaml
└── infrastructure
    └── controllers
        ├── argocd-operator
        │   ├── configmap-patch.yaml
        │   ├── kustomization.yaml
        │   ├── namespace.yaml
        │   └── patch-argocd-server-annotations.yaml
        ├── argocd-operator-apps
        │   ├── applications.yaml
        │   ├── kustomization.yaml
        │   ├── projects.yaml
        │   └── repositories.yaml
        ├── csi-driver-smb
        │   ├── csi-driver-smb.yaml
        │   └── kustomization.yaml
        ├── external-secrets
        │   ├── external-secrets-operator.yaml
        │   └── kustomization.yaml
        ├── external-secrets-secret
        │   ├── cluster-secret-store.yaml
        │   └── kustomization.yaml
        ├── tailscale-operator
        │   ├── kustomization.yaml
        │   └── tailscale-operator.yaml
        ├── tailscale-operator-secrets
        │   ├── kustomization.yaml
        │   └── tailscale-operator-exteral-secret.yaml
        └── traefik
            ├── kustomization.yaml
            └── traefik-ext-conf.yaml

Now we need some explanation, right?

Flux configuration

clusters
└── cluster0
    └── flux-system
        ├── gotk-components.yaml
        ├── gotk-sync.yaml
        ├── infrastructure.yaml
        └── kustomization.yaml

Here we have the configuration of our cluster, which is an awesome idea. In one repository we can have configurations for multiple clusters, based on provider, environment, or location, and manage them in a very simple way. Then we have regular flux-system files, so flux-system/gotk-components.yaml and flux-system/gotk-sync.yaml. Next, let's talk about my simple Kustomization file

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - gotk-components.yaml
  - gotk-sync.yaml
  - infrastructure.yaml

This just tells Flux, that after bootstrapping itself, install
manifests based on infrastructure.yaml file. So let's take a look at the most crucial part of the config.

---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: csi-driver-smb
  namespace: flux-system
spec:
  interval: 1h
  retryInterval: 1m
  timeout: 5m
  sourceRef:
    kind: GitRepository
    name: flux-system
  path: ./infrastructure/controllers/csi-driver-smb
  prune: true
  wait: true
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: external-secrets
  namespace: flux-system
spec:
  interval: 1h
  retryInterval: 1m
  timeout: 5m
  sourceRef:
    kind: GitRepository
    name: flux-system
  path: ./infrastructure/controllers/external-secrets
  prune: true
  wait: true
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: external-secrets-secret
  namespace: flux-system
spec:
  interval: 1h
  retryInterval: 1m
  timeout: 5m
  sourceRef:
    kind: GitRepository
    name: flux-system
  path: ./infrastructure/controllers/external-secrets-secret
  dependsOn:
    - name: external-secrets
  prune: true
  wait: true
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: tailscale-operator
  namespace: flux-system
spec:
  interval: 1h
  retryInterval: 1m
  timeout: 5m
  sourceRef:
    kind: GitRepository
    name: flux-system
  path: ./infrastructure/controllers/tailscale-operator
  dependsOn:
    - name: external-secrets-secret
  prune: true
  wait: true
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: tailscale-operator-secret
  namespace: flux-system
spec:
  interval: 1h
  retryInterval: 1m
  timeout: 5m
  sourceRef:
    kind: GitRepository
    name: flux-system
  path: ./infrastructure/controllers/tailscale-operator-secrets
  dependsOn:
    - name: external-secrets
  prune: true
  wait: true
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: traefik
  namespace: flux-system
spec:
  interval: 1h
  retryInterval: 1m
  timeout: 5m
  sourceRef:
    kind: GitRepository
    name: flux-system
  path: ./infrastructure/controllers/traefik
  prune: true
  wait: true
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: argocd-operator
  namespace: flux-system
spec:
  interval: 1h
  retryInterval: 1m
  timeout: 5m
  sourceRef:
    kind: GitRepository
    name: flux-system
  path: ./infrastructure/controllers/argocd-operator
  dependsOn:
    - name: tailscale-operator
  prune: true
  wait: true
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: argocd-apps
  namespace: flux-system
spec:
  interval: 1h
  retryInterval: 1m
  timeout: 5m
  sourceRef:
    kind: GitRepository
    name: flux-system
  path: ./infrastructure/controllers/argocd-operator-apps
  dependsOn:
    - name: argocd-operator
  prune: true
  wait: true

As you can see I heavily rely on dependsOn command. That is due Flux and Kubernetes archtecture design. When we're applying a Kustomization, we do not control the order of creating the resources. Of course, if we put in one file, service, ingress, and deployment, Kubernetes will know how to handle that. The problem is when we deploy an application that has dependencies on each other, that is not what Kubernetes understands. So we need to specify it directly. In my example csi-driver-smb, external-secrets, and traefik can be installed in parallel, but then we have some relations. At the end, we're installing Argo's app-of-apps, which requires in general all previous components, besides traefik - I'm routing my traffic through Tailscale there, not via the Internet.

Now you can think:

App-of-apps? You said infra only!

That is right, my logic was quite complex, to be honest here. Do you remember, when I wrote about use-case? Bootstraping the whole cluster so that I will be able to work with it fast. Apps are part of the overall cluster at the end. So my app-of-apps definition was very simple:

infrastructure/controllers/argocd-operator-apps
├── applications.yaml
├── kustomization.yaml
├── projects.yaml
└── repositories.yaml

Let's dive into it one, one by one.

applications.yaml

 ---
 apiVersion: argoproj.io/v1alpha1
 kind: Application
 metadata:
   name: self-sync
   namespace: argocd
 spec:
   syncPolicy:
     automated:
       selfHeal: true
   project: non-core-namespaces
   source:
     repoURL: https://codeberg.org/3sky/argocd-for-home
     targetRevision: HEAD
     path: app-of-apps
   destination:
     server: https://kubernetes.default.svc
     namespace: argocd

That is a very simple definition of my main Application that controls all apps running on my cluster. Nothing very special here, I was experimenting with codeberg, but let's talk about that later.

kustomization.yaml

  ---
  apiVersion: kustomize.config.k8s.io/v1beta1
  kind: Kustomization

  namespace: argocd
  resources:
    - repositories.yaml
    - projects.yaml
    - applications.yaml

Kustomization is simple, just ordered in the logic path.

Repository -> Projects -> Application

projects.yaml

  ---
  apiVersion: argoproj.io/v1alpha1
  kind: AppProject
  metadata:
    name: non-core-namespaces
    namespace: argocd
  spec:
    description: Allow argo deploy everywhere
    sourceRepos:
      - 'https://codeberg.org/3sky/argocd-for-home'
    destinations:
      - namespace: '*'
        server: https://kubernetes.default.svc
    namespaceResourceWhitelist:
      - group: '*'
        kind: '*'
    clusterResourceWhitelist:
      - group: '*'
        kind: '*'

As I'm creating namespaces as part of my application setup, permission granted to Argo's Projects are very wide. In this case, I just trust the GitOps process.

repositories.yaml

  apiVersion: external-secrets.io/v1beta1
  kind: ExternalSecret
  metadata:
    name: gitops-with-argo-secret
    namespace: argocd
  spec:
    refreshInterval: 6h
    secretStoreRef:
      name: doppler-auth-argocd
      kind: ClusterSecretStore
    target:
      name: gitops-with-argo
      creationPolicy: Owner
      template:
        type: Opaque
        metadata:
          labels:
            argocd.argoproj.io/secret-type: repository
        data:
          type: git
          url: https://codeberg.org/3sky/argocd-for-home
          username: 3sky
          password: "{{ .password }}"
    data:
      - secretKey: password
        remoteRef:
          key: CODEBERG_TOKEN

This is probably the most interesting part of the configuration. As we're unable to inject our password directly into argocd.argoproj.io/secret-type: repository, which is a regular Kubernetes Secret. We need to generate the whole object
with ESO - details can be found here.

And that's it. Now let's talk about the actual bootstrap process.

Bootstraping environment

Spin up infrastructure with Terraform.

terraform apply -var="local_ip=$(curl -s ifconfig.me)"

Installing K3S with local fork of k3s-ansible.

ansible-playbook playbooks/site.yml -i inventory.yml

Load KUBECONFIG

export KUBECONFIG=/home/kuba/.kube/config.hetzner-prod
kubectl config use-context k3s-ansible

Doppler configuration (we need an initail secret somewhere)

kubectl create namespace external-secrets
kubectl create secret generic \
  -n external-secrets doppler-token-argocd \
  --from-literal dopplerToken=""

Bootstrap the cluster

flux bootstrap github \
  --owner=3sky \
  --repository=flux-at-home \
  --branch=main \
  --path=./clusters/cluster0 \
  --personal

Where GitHub is supported natively (via GitHub Apps). Solutions like Codeberg needs a more direct method.

flux bootstrap git \
  --url=ssh://git@codeberg.org/3sky/flux-at-home.git \
  --branch=main   \
  --path=./clusters/cluster0   \
  --private-key-file=/home/kuba/.ssh/id_ed25519_git

Get ArgoCD password if needed

kubectl --namespace argocd \
  get secret argocd-initial-admin-secret \
  -o json | jq -r '.data.password' | base64 -d

Summary

With this set of commands, I'm able to set fresh clusters, with Hetzner or whatever in literally 7 minutes. Starting from installing k3s to having applications exposed via Cloudflare tunnel, or Tailscale tailnet. To be honest I'm really satisfied with the result of this exercise. Besides a long-running cluster for my self-hosted apps. I can quite easily, fast, and with low costs test new versions of ArgoCD, or smb-driver operator, without harming my current setup! And that is great, to be honest. Especially for self-hosting, where it's good to have apps that are working in let's say production mode, even if there is only one user.

External Secret Operator with Doppler

Jakub Wołynko — Tue, 15 Apr 2025 07:18:10 +0000

Last time we configured our cluster step by step maybe without public code yet, but someday I will publish it. Probably when it will be smooth enough to share. Nevertheless, we have a working cluster. Today I will focus on connecting the External Secret Operator with Doppler. So let's introduce today's stars.

External Secret Operator

External Secrets Operator is a Kubernetes operator that
Integrates external secret management systems like AWS Secrets Manager,
HashiCorp Vault, Google Secrets Manager, Azure Key Vault,
IBM Cloud Secrets Manager, CyberArk Conjur,
Pulumi ESC and many more. The operator reads information from
external APIs and automatically inject the values into a Kubernetes Secret.

Link

Doppler

Doppler integrates with popular CI/CD tools
and frameworks, making it easy to automate secrets
management within your development workflow.

Link

Fast guide over platform

Create a project

Adding new project is very simple, and easy.
Just click on + button near to the Projects text.
Manage stages

Here we can add, delete, or modify stages. At the beginning, I would like to recommend using only one stage.
Adding secrets

Then after clicking on the stage name, we can add next secret with two-click.
Getting a token

The last action to do is token generation. To archive it, just go to *Access tab, then generate a token, that will have access to particular config.

How to use this duet

First, we need to understand how it works. In my cluster, it looks
as shown in the picture below:

As you may see, we need to have one "static secret" in our cluster, we can make it from CLI directly or with the usage of sops, however, the effect will be the same, where we need to start. I decided that my workstation terminal session is a good place to create the main secret.

When we have our token, we can call Doppler and retrieve needed secrets. In my case, I'm using one Doppler's project for everything and all namespaces have access to my token, but we can restrict it on the RBAC level if needed. Or not, that is why we're using ESO.

Retrieving secrets

In my opinion, we have at least 2 modes of getting credentials, that should meet all our needs.

Getting all prefixed secrets.

---
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: hello
  namespace: hello
  labels:
    app.kubernetes.io/component: externalsecret
    app.kubernetes.io/part-of: hello
    app.kubernetes.io/managed-by: Kustomize
spec:
  refreshInterval: 6h
  secretStoreRef:
    name: doppler-auth-argocd 
    kind: ClusterSecretStore
  dataFrom:
    - find:
        path: HELLO_

With this code, we're fetching all secrets with prefix HELLO_, into a single secret with the name hello. And that's it. If you want to check the result of your integration just run:

kubectl get secrets hello \
    -o jsonpath="{.data.HELLO_SECRET}" \ 
    -n hello | base64 --decode

Getting particular secret and putting them into a specific order

---
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: hello
  namespace: hello
  labels:
    app.kubernetes.io/component: externalsecret
    app.kubernetes.io/part-of: hello
    app.kubernetes.io/managed-by: Kustomize
spec:
  refreshInterval: 6h
  secretStoreRef:
    name: doppler-auth-argocd 
    kind: ClusterSecretStore
  target:
    name: hello-secret
    creationPolicy: Owner
    template:
      data:
        APP_SECRET: "{{ .HELLO_SECRET }}"
        INIT_PASSWORD: "{{ .HELLO_INIT_PASSWORD }}"
  dataFrom:
    - find:
        path: HELLO_

This will create secret/hello-secret with a specific object structure. When it could be helpful? I will show you 3 interesting examples, that you could use in your setup.

Adding Cloudflare tunnel config file

Cloudflare tunnels require providing data in a specific format.
It's a JSON string, looking similar to:

{
  "AccountTag": "f3cc889ae810",
  "TunnelSecret": "SVD/O0sU+VtDlH023aBcToG0dMk=",
  "TunnelID": "8127899-c332-482c-91aa-d1210a479658"
}

And you need to use exactly this as a file-based secret. Then you have two option get your prefixed secrets into one object, then credentials.json into second, or use templates.

---
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: hello
  namespace: hello
  labels:
    app.kubernetes.io/component: externalsecret
    app.kubernetes.io/part-of: hello
    app.kubernetes.io/managed-by: Kustomize
spec:
  refreshInterval: 6h
  secretStoreRef:
    name: doppler-auth-argocd 
    kind: ClusterSecretStore
  target:
    name: hello-secret
    creationPolicy: Owner
    template:
      data:
        credentials.json: "{{ .HELLO_TUNNEL }}"
        APP_SECRET: "{{ .HELLO_APP_SECRET }}"
        DB_SECRET: "{{ .HELLO_DB_SECRET }}"
        DATABASE_URL: "{{ .HELLO_DB_URL }}"
  dataFrom:
    - find:
        path: HELLO_
---

Building ArgoCD repository secret.

If you have worked with ArgoCD in the past, there is a chance, that you know that problem is not trivial. You need to use build object with type secret, with repository URL, user, and secret. As you can't have a nested secret (secret ref in another secret), you are forced to keep the whole file secret. Fortunately, you can use ESO templates.

---
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: gitops-with-argo-secret
  namespace: argocd
spec:
  refreshInterval: 6h
  secretStoreRef:
    name: doppler-auth-argocd 
    kind: ClusterSecretStore
  target:
    name: gitops-with-argo
    creationPolicy: Owner
    template:
      type: Opaque
      metadata:
        labels:
          argocd.argoproj.io/secret-type: repository
      data:
        type: git
        url: https://github.com/3sky/argocd-for-home
        username: 3sky
        password: "{{ .password }}"
  data:
    - secretKey: password
      remoteRef:
        key: GITHUB_TOKEN

At the end, simple, direct Tailscale mapping

---
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: operator-oauth-secret
  namespace: tailscale
spec:
  refreshInterval: 6h
  secretStoreRef:
    name: doppler-auth-argocd 
    kind: ClusterSecretStore
  target:
    name: operator-oauth
    creationPolicy: Owner
    template:
      data:
        client_id: "{{ .client_id }}"
        client_secret: "{{ .client_secret}}"
  data:
    - secretKey: client_id
      remoteRef:
        key: TAILSCALE_CLIENT_ID
    - secretKey: client_secret
      remoteRef:
        key: TAILSCALE_CLIENT_SECRET

Summary

Based on my testing and usage of this integration, so far so good. Stable, not very resource-consuming, we can limit sync intervals. Doppler as mentioned at the beginning is free for small projects, or individual users. However, using ESO with popular enterprise solutions like Hashicorp Vault should be very similar. Maybe I will check it someday.

I hope you like this kind of article, smaller, focused on a particular problem or integration. If not, let me know.

Again self-hosting! on k3s

Jakub Wołynko — Tue, 15 Apr 2025 07:07:04 +0000

Over some time I was really happy with my podman + ansible setup. It was great, but do you know what wasn’t such great? Deployment rollbacks. It all started with linkwarden. On my miniflux, I received a notification - that a new minor release is ready.

You can use GitHub repos as RSS links and received
notification about new releases. If you're using miniflux
just past https://github.com/ansible/ansible/releases.atom

Let's say version 2.8.0. When the time came, I just changed my variable and executed ansible-playbook command. Then after 3 minutes, my uptime-kuma started screaming, and saying that my app was dead. Initially, I ignored that, due to my experience with podman. During the service restart it makes an app unavailable, then downloads the new image and tries to run it. If the package is big, it could take a while. It wasn't great, but that is what could be acceptable for self-hosting service. Unfortunately, the main problem was with something different. Maintainers of Linkwarden sometimes inform users about a new release, however, it's only delivered as an rpm/deb package. The container image could be still in the building, or the building process could fail. It doesn't matter, sometimes my typo could occur as well. Then podman can't handle that, nor Ansible as it just restarts the service. Podman on another hand just stops service and does not roll back to the old version automatically. So sometimes it becomes annoying as I need to log into the server with ssh and manually fix it. That is why I decided to switch to some Kubernetes distribution.

k3s

So why is the based version of small Kubernetes supported by Suse?
Hmm mostly due to popularity, stability, and small resource consumption. It was good enough to run on a single host, and it's shipped as just a binary. This fits my needs, a small, easy-to-use package with build-in ingress, local-path storage provider, and flannel that implements Kubernetes CNI. Here is also a really nice
project for installation and configuration base cluster, which I used for my setup.

The migration process

I did not have much time for migration, as I really enjoy spending time with my family, also with a full-time job it is just hard to put in the calendar. That is why I decided to join the "5 AM Club", and start re-learning Kubernetes again. I have a full log of my activities with dates, but probably that is not what you could be interested in. Let's say my regular everyday process look like this:

Apply terraform code
```
cd hetzner
terrafrom apply
```

Run k3s-ansible project, (check IP in inventory.yaml).

cd k3s-ansible
ansible-playbook playbooks/site.yml -i inventory.yml

With ready cluster update KUBECONFIG.

export KUBECONFIG=~/.kube/config.new
kubectl config use-context k3s-ansible

Patch k3s setup with traefik config.

cd cluster-config
k apply -f traefik-ext-conf.yaml

Create External Secret Operator main token for doppler

kubectl create namespace external-secrets
kubectl create secret generic \
    -n external-secrets \
    doppler-token-argocd \
    --from-literal dopplerToken="dp.st.xx"

Install EOS operator

helm install external-secrets \
   external-secrets/external-secrets \
    -n external-secrets \
    --create-namespace \
    --set installCRDs=true

Check if the webhook is up and running (sometimes is not)

k logs -l app.kubernetes.io/name=external-secrets-webhook -n external-secrets

If yes, apply the overlay and create ClusterSecretStore
```
cd ESO/
kubectl apply -k overlay/
```
Install tailscale
```
cd tailscale
kubectl apply -k .
```

Install argo

cd argocd
kubectl apply -k base/
kubectl apply -k overlay/

Get Argo init admin password

kubectl --namespace argocd get \
    secret argocd-initial-admin-secret \
    -o json \
    | jq -r '.data.password' \
    | base64 -d

So far so good. Now let's break down used services.

Operators

I started by extending my traefik configuration to be able to handle regular requests from the internet. If you decide to do it,
as well please be aware that k3s for today (07-Feb-2025) is using version 2+, not 3+, which is why not everything straight from documentation will work. For example, my code is:

kind: HelmChartConfig
metadata:
  name: traefik
  namespace: kube-system
  # we're still on with k3s https://github.com/traefik/traefik-helm-chart/blob/v27.0.2/traefik/values.yaml
spec:
  valuesContent: |-
    additionalArguments:
      - "--certificatesresolvers.letsencrypt.acme.email=mark@example.com"
      - "--certificatesresolvers.letsencrypt.acme.storage=/data/acme.json"
      - "--certificatesresolvers.letsencrypt.acme.httpchallenge.entrypoint=web"
      # - "--certificatesresolvers.letsencrypt.acme.caServer=https://acme-staging-v02.api.letsencrypt.org/directory"

With that, and duckdns domain (Hetzner,
very often gave you the same IP address. Thanks folks!). I was
able to expose my service directly to the internet with the usage HTTPS.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: argocd
  namespace: argocd
  annotations:
    traefik.ingress.kubernetes.io/router.entrypoints: "websecure"
    traefik.ingress.kubernetes.io/router.tls: "true"
    traefik.ingress.kubernetes.io/router.tls.certresolver: "letsencrypt"
    traefik.ingress.kubernetes.io/service.serversscheme: "h2c"
spec:
  ingressClassName: "traefik"
  tls:
    - hosts:
        - nginx997.duckdns.org
  rules:
  - host: nginx997.duckdns.org
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: nginx
            port:
              number: 80

Then we have a challenge called:

How to store secrets with the usage of external sources?

I decided to use two products: External Secret Operator, and Doppler. In the beginning, I thought about Bitwarden's "not-so-new" Secret Manager, however, after a short investigation, the product seems to be not so well-supported by the ESO, which IMO is useful as it allows me to have one cluster-wide secret for getting secrets from external sources, which is great.

Doppler and ESO combo requires another post, so check my website from time to time.

Then I wanted to add Tailscle which besides being a "best in class VPN" for the homelabbers, allows you to add k8s services directly into your tailnet. What does it mean? The Tailscale operator allows you to access your k8s applications only when you are logged into your private
network (tailnet), with the usage of your domain for ended with ts.net. You can configure it in two ways on the resource side, with ingress or with service annotation.

Ingress

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: hello
  namespace: hello
spec:
  ingressClassName: tailscale
  tls:
    - hosts:
        - hello.john-kira.ts.net
  rules:
    - host: hello.john-kira.ts.net
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: nginx
                port:
                  number: 80

Service

---
apiVersion: v1
kind: Service
metadata:
  name: hello
  annotations:
    tailscale.com/expose: "true"
    tailscale.com/tailnet-fqdn: "hello.john-kira.ts.net"
    tailscale.com/hostname: "hello"
spec:
  ports:
    - name: http
      port: 80
      targetPort: 80
      protocol: TCP
  selector:
    app: nginx

Also please be aware that service by default exposes your service into tailnet over HTTP, where ingress provides a TLS certification as well.

Nice, at this point I was able to access my public service, as well as internal. What was the next step? ArgoCD.

Deploying ArgoCD was simple. During the first iteration, I decided to split my repo into base and overlay folders. The first directory contains the file for deploying an instance of Argo, exposing it to the tailnet. To archive it I just created a simple kustomization.yaml file as below:

---
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

namespace: argocd
resources:
- namespace.yaml
- https://raw.githubusercontent.com/argoproj/argo-cd/v2.13.2/manifests/install.yaml

# add --insecure flag to deployment, to avoid 307 redirection loop
patches:
- target:
    kind: ConfigMap
    name: argocd-cmd-params-cm
  path: configmap-patch.yaml
- target:
    kind: Service
    name: argocd-server
  path: patch-argocd-server-annotations.yaml

Then two patches were:

patch-argocd-server-annotations.yaml

apiVersion: apps/v1
kind: Service
metadata:
  name: argocd-server
  annotations:
    tailscale.com/expose: "true"
    tailscale.com/tailnet-fqdn: "argo.john-kira.ts.net"
    tailscale.com/hostname: "argo"

configmap-patch.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-cmd-params-cm
  namespace: argocd
data:
  server.insecure: "true"

Then my overlay directory contains only Argo's objects definitions:

overlay
├── applications.yaml
├── kustomization.yaml
├── projects.yaml
└── repositories.yaml

Summary

That was my first iteration of the "5 AM Club" Kubernetes migration. It takes me a bit longer than one or two mornings. Based on my notes it was around 7 days of work, ~1h per day. Not bad at all, however, there were a few things that are missing and which I should improve to make my setup much more flexible, stable, and easier to test. Where "to test" I understand deploying a new version of the operator and watching how the cluster is burning. Or not, I hope rather not.

With that in mind, thanks for your time, hope you enjoy reading it. If you would like to reach me or you have questions please use about page infos.

Self-hosting with Podman

Jakub Wołynko — Tue, 15 Apr 2025 06:53:31 +0000

I've been a self-hoster for a while. The adventure started with regular mani-pc manufactured by HP. 32G of RAM, Intel gen 10, and 1T HDD drive. However, as long as it was a great experience at the beginning, with time it became a challenge. My stack was built with portainer and a bunch of docker-compose files. It leads to specific issues: with portainer, you don't own the Compose files, they are living inside your tool, but nowhere on the filesystem or git repo.

update Feb-2025: portioner supports GitOps

Additionally at some point people behind this product decided to change the licensing model, and allow the use of community editions for up to 5 nodes. It wasn't my case, but that pushed me to use something more independent. So I started using dockge, then added another service for docker logs, version monitor, and keeps adding applications that are fun to use, for example homebox or bookstack. It was fun until I released the cost of energy and maintenance effort need to keep it running, at my home. Every internet issue, or power issue takes my setup down. Maybe it was not happening very often, but when I wasn’t home, and the hardware was down, there was no chance to fix it remotely. And I started relaying on that service. That is why I simply decided
to migrate to hetzner, and podman at the same time, and use remote NFS. However, let's start from the beginning.

Why Hetzner

As I'm an AWS Community Builder, every renewal cycle I receive $500 for AWS services. I was getting an EC2 instance on ARM (yes, I decided to switch CPU architecture too) with 4 cores and 8GB of RAM, which cost more than $500 per year. Since we're talking about a server running 24/7, with additional VPC, EBS, etc., the estimated cost was just $111.744 per month. For EC2 only!

Hetzner is much more affordable. A standard AmpereOne VM, with backups, a public IP, and 80GB of SSD, was about $7 per month. Then I added a 1TB storage box for an additional $3. Based on complex math, my setup was only 10x cheaper than what AWS could provide.

Note: We're talking about a constantly running server, without the need for scalability or high availability.

Using Hetzner is pretty simple. If you know how to use Terraform and AWS, the German company provider is less complex. For example, spinning up one VM with a public IP address and backup enabled is just:

terraform {
  required_providers {
    hcloud = {
      source  = "hetznercloud/hcloud"
      version = "~> 1.49.1"
    }
  }
  required_version = "~> 1.9.8"
}


provider "hcloud" {
  token = var.hcloud_token
}

variable "hcloud_token" {
  sensitive = true
  type      = string
}


variable "local_ip" {
  type    = string
  default = "79.184.235.150"
  #default = "31.61.169.52"
}

resource "hcloud_primary_ip" "box_ip" {
  name          = "box-ip"
  datacenter    = "fsn1-dc14"
  type          = "ipv4"
  assignee_type = "server"
  auto_delete   = true
  labels = {
    "arch" : "arm64",
    "managed_by" : "terraform",
    "env": "prod",
    "location": "de"
  }
}


resource "hcloud_firewall" "box_fw" {
  name = "box-firewall"

  rule {
    direction  = "in"
    protocol   = "tcp"
    port       = "22"
    source_ips = [var.local_ip]
  }
  labels = {
    "arch" : "arm64",
    "managed_by" : "terraform",
    "env": "prod",
    "location": "de"
  }
}

resource "hcloud_server" "box" {
  name        = "box"
  image       = "centos-stream-9"
  server_type = "cax21"
  datacenter  = "fsn1-dc14"
  ssh_keys    = ["mbp@home"]
  backups     = true
  labels = {
    "arch" : "arm64",
    "managed_by" : "terraform",
    "env": "prod",
    "location": "de"
  }
  public_net {
    ipv4_enabled = true
    ipv4         = hcloud_primary_ip.box_ip.id
    ipv6_enabled = false
  }

  firewall_ids = [hcloud_firewall.box_fw.id]
}

output "box_public_ip" {
  value = hcloud_server.box.ipv4_address
}

output "ssh_box" {
  value = "ssh -i ~/.ssh/id_ed25519_local root@${hcloud_server.box.ipv4_address}"
}

Then we can execute our code with a simple:

terraform apply -var="local_ip=$(curl -s ifconfig.me)"

To list accessible images, use hcloud image list.

So, when we have our CentOS box, let's install some baseline packages there.

Ansible

The standard tool for this will be Ansible - a simple, stable, and solid product. 100% open source. For setting up my server, I decided to write a custom role. The structure of my role's task folder is simple but, in my opinion, requires some explanation.

Tasks's name started with 01 and 02 are core services.
Tasks's name started with 03 are responsible for packages.
Tasks's name started with 1 are application services.

tasks
├── 010_shost.yml
├── 020_ssh.yml
├── 030_packages.yml
├── 035_tailscale.yml
├── 036_storagebox.yml
├── 100_containers.yml
├── 101_linkwarden.yml
├── 102_miniflux.yml
├── 103_umami.yml
├── 104_internal.yml
├── 105_immich.yml
├── 106_jellyfin.yml
└── main.yml

main.yml is a role control point; I have tags here and custom

logic:

# tasks file for roles/hetzner
- name: Ensure that shost exists, it's a main user, and root cannot access the server
  tags:
    - baseline
  ansible.builtin.import_tasks:
    file: 010_shost.yaml

- name: Ensure that ssh config is correct
  tags:
    - baseline
  ansible.builtin.import_tasks:
    file: 020_ssh.yaml

- name: Ensure that needed packages are installed
  tags:
    - packages
  ansible.builtin.import_tasks:
    file: 030_packages.yaml

- name: Ensure that Tailscale is installed if needed
  tags:
    - vpn
  ansible.builtin.import_tasks:
    file: 035_tailscale.yaml

- name: Ensure that StorageBox was attached
  tags:
    - storage
  when:
    - hetzner_storagebox_enabled
  ansible.builtin.import_tasks:
    file: 036_storagebox.yaml

- name: Ensure that containers have latest configuration
  tags:
    - never
    - containers
  ansible.builtin.import_tasks:
    file: 100_containers.yaml

Then, every new application has a dedicated playbook, which will look like 101_linkwarden.yml.

- name: Supply system with Linkwarden network configuration
  notify:
    - Restart Linkwarden
  ansible.builtin.template:
    src: linkwarden.network.j2
    dest: /home/shost/.config/containers/systemd/linkwarden.network
    mode: '0700'
    owner: shost
    group: shost

- name: Supply system with Linkwarden service
  notify:
    - Restart Linkwarden
  ansible.builtin.template:
    src: linkwarden.service.j2
    dest: /home/shost/.config/systemd/user/linkwarden.service
    mode: '0700'
    owner: shost
    group: shost

- name: Supply system with Linkwarden Postgresq config
  notify:
    - Restart Linkwarden
  ansible.builtin.template:
    src: linkwarden-postgresql.container.j2
    dest: /home/shost/.config/containers/systemd/linkwarden-postgresql.container
    mode: '0700'
    owner: shost
    group: shost

- name: Supply system with Linkwarden server
  notify:
    - Restart Linkwarden
  ansible.builtin.template:
    src: linkwarden-app.container.j2
    dest: /home/shost/.config/containers/systemd/linkwarden-app.container
    mode: '0700'
    owner: shost
    group: shost

- name: Supply system with Linkwarden Tunnel
  notify:
    - Restart Linkwarden
  ansible.builtin.template:
    src: linkwarden-tunnel.container.j2
    dest: /home/shost/.config/containers/systemd/linkwarden-tunnel.container
    mode: '0700'
    owner: shost
    group: shost

You may be wondering why I'm not using docker-compose and have a lot of systemd services instead. Let me explain it.

Podman

By default, Podman supports rootless containers. What does it mean? Basically, you don't need to be root or a member of the docker group to run the container. All the magic happens inside the user namespace and only has access to the user's data. As an extra feature, we have SELinux context on files as another security layer. However, as good as the community.docker.docker_compose_v2 module is, there is no Podman equivalent. Folks responsible for the project are saying that you should use quadlets, not composes. WTF are quadlets? Generally speaking, they are systemd services that live in a user namespace and orchestrate Podman containers. Unfortunately, one by one. Wait, what? Yes, the user needs to write a new service per container, and network, and service dependencies. Sounds like fun, isn't it?

There is a project on the internet that allows you to convert your docker-compose directly into quadlets. However, I will show you my systemd services, which could be helpful.

linkwarden.service.j2 is a dummy service that allows me to control the whole application with one service.

[Unit]
Description=Linkwarden

[Service]
Type=oneshot
ExecStart=/bin/true
RemainAfterExit=yes

[Install]
WantedBy=basic.target

linkwarden.network.j2 is a simple definition of my separated network, only for this particular app.

[Unit]
Description=Linkwarden - Network
PartOf=linkwarden.service

[Network]

linkwarden-app.container.j2 is the main service for my app. As I'm using Ansible and Jinja2, avoiding hardcoded credentials is easy (use sops).

[Unit]
Description=Linkwarden - Server
PartOf=linkwarden.service
After=linkwarden.service
After=linkwarden-network.service
After=linkwarden-postgresql.service

[Container]
Image=ghcr.io/linkwarden/linkwarden:v{{ hetzner_linkwarden_app_version }}
ContainerName=linkwarden-app
Network=systemd-linkwarden
Volume=linkwarden-data:/data/data
LogDriver=journald

Environment="DATABASE_URL=postgresql://postgres:{{ linkwarden_postgresql_password }}@linkwarden-postgresql:5432/postgres"
Environment=NEXTAUTH_SECRET={{ linkwarden_next_auth_secret }}
Environment=NEXTAUTH_URL=http://localhost:3000/api/v1/auth
Environment=NEXT_PUBLIC_DISABLE_REGISTRATION=true

[Service]
Restart=always

[Install]
WantedBy=linkwarden.service

As you may have noticed, specifying the volume is very straightforward. The usage of After/PartOf is needed, and what is tricky is that the whole image path is required, as Podman could have issues finding images like linkwarden/linkwarden.

linkwarden-postgresql.container.j2 - database service

[Unit]
Description=Linkwarden - Postgresql
PartOf=linkwarden.service
After=linkwarden.service
After=linkwarden-network.service

[Container]
Image=docker.io/library/postgres:16-alpine
ContainerName=linkwarden-postgresql
Network=systemd-linkwarden
Volume=linkwarden-postgresql:/var/lib/postgresql/data
LogDriver=journald

Environment="POSTGRES_PASSWORD={{ linkwarden_postgresql_password }}"

[Service]
Restart=always

[Install]
WantedBy=linkwarden.service
linkwarden-tunnel.container.j2 - Cloudflare tunnels

[Unit]
Description=Linkwarden - Server
PartOf=linkwarden.service
After=linkwarden.service
After=linkwarden-postgresql.service
After=linkwarden-network.service
Requires=linkwarden-app.service

[Container]
ContainerName=linkwarden-tunnel
Exec=tunnel --no-autoupdate run
Image=docker.io/cloudflare/cloudflared:{{ hetzner_tunnel_version }}
Network=systemd-linkwarden
Volume=linkwarden-tunnel:/etc/cloudflared
LogDriver=journald

Environment="TUNNEL_TOKEN={{ linkwarden_tunnel_token }}"

[Service]
Restart=always

[Install]
WantedBy=linkwarden.service

I like the idea of tunnels, even if all of them are commercial software. They allow me to expose my service to the internet without needing to set up NGINX or Caddy, and more importantly, hardening them.

Summary

So far so good. The solution seems complex, but after the initial setup, it is very secure and stable. The regular path of upgrading my services is changing the version of the images I'm using in the group_vars file.

$ diff --git a/apps.yaml b/apps.yaml
index 63b65b2..e08c7a3 100644
--- a/apps.yaml
+++ b/apps.yaml
@@ -1 +1 @@
-hetzner_linkwarden_app_version: 2.8.3
+hetzner_linkwarden_app_version: 2.9.3

Then just running:

$ ansible-playbook \
  selfhost-hetzner.yaml \
  --tags containers -u shost

That's it. What am I thinking after 4 months of using Hetzner in production? I'm very happy with them. The price is unbeatable, it's stable, and the user experience is very high level. For a project like this one, I can't recommend it more.

Ah, a few words about the AmpereOne CPUs. For the services I'm using, there is no problem with using ARM64 binaries.

Miniflux
Jellyfin
Immich
Linkwarden
N8N
Caddy
Actual-budget
uptime-kuma
ghost

All of them are running very well on an ARM64 CPU, 4 CPU cores and 8GB of RAM to be precise. This is probably due to the popularity of Raspberry Pi in the self-hosting landscape. So yes, if you have a chance to use ARM CPUs, let's give them a spin. It will be cheaper, probably more efficient, and you could give yourself an ‘innovative soul’ award.

OneDev on ECS - How to host your own instance of OneDev on AWS

Jakub Wołynko — Mon, 16 Sep 2024 06:00:00 +0000

Welcome

At the beginning of September, I saw a post from Johannes Koch about [self-hosting CodeCommit alternative (https://lockhead.info/index.php/2024/09/03/a-self-hosted-codecommit-alternative/). Then I realised, that I'm using my home, not such a popular, but very nice and solid git repository system. To be more precise, much more than a git server. Let's welcome OneDev.

Git server with CI/CD, kanban, and packages.

They also support repo tags, and groups like GitLab.
The project is also very resource-friendly, and I've been using it for a while on my home server.

And to be honest no issues so far, however, I store only code there. I don't have an opinion about package registry, CICD, and Kanban, but if you need it, try it!

Also if you're thinking about hosting on AWS apps that you’re using, instead of using regular units under the desk because of the cost of energy in your current location is very high, or you are just not sure if the hobby is fits your personality. This blog is a great place to be, as I'm starting "self-host on AWS” series.

Why host a git server on your own?

The main point here is privacy. Sometimes laziness as well.
Let's imagine the situation when you push accidentally AWS keys into the repository or SSH key. If you’re using GitHub, after a while you will get a notification that one of your repositories contains a secret, so you should change it immediately (don't ask me how I know this). There is always a change, the someone already
started using it, so your AWS bills are going higher with every second. If it is a private internal git server, nobody cares, as you trust your coworkers, right? Nobody will judge your code, even if you're using, <put random JS framework name here>.
However, the real reason is code ownership. Some organisations would like to fully own their code, or at least host it in Europe. Now we, how to answer the why question. Let's now focus on the how question.

Implementation

As I already said, the initial point of implementation was my docker-compose file.

---
name: onedev
---
volumes:
  data:
    name: onedev_data
---
services:
  onedev:
    image: 1dev/server:11.0.9
    container_name: onedev
    restart: always
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - data:/opt/onedev
    ports:
      - 6610:6610
      - 6611:6611

On top of it, I have Caddy reverse proxy config:

{
    log default {
        output stdout
        format json
        exclude http.handlers.reverse_proxy
    }
}

*.pinkwall.cc {
  tls {
    dns cloudflare M8zaR6d1la_SI1vggnrnl3-Fhfa11359bnn31
  }

  @git host git.pinkwall.cc
       handle @git {
               reverse_proxy 100.16.152.33:6610
       }
}

An SSH config, as Caddy, can't reverse proxy SSH.

git.pinkwall.cc
  PreferredAuthentications publickey
  IdentitiesOnly=yes
  IdentityFile ~/.ssh/id_ed25519_local
  User 3sky
  Port 6611

So when I started thinking about the AWS-based design, I created a
diagram, initially on paper, but the digital one looks pro.

As you can see, I'm using NLB, that's due to the fact, that I wanted to operate on layer 4 (TCP), not L7 (HTTP). Why? Because using git over ssh is way more smoother.
EFS as persistent storage. As we need to store our data, right?
EFS access points have root permission, inside the volume, that is because OneDev expects it.
ECS with Fargate, to make it cost efficient as possible.
We're using standard 2-tier subnet architecture with small CIDR ranges.
- We're not interested in having ECS Cluster exposed directly to the Internet.
- We do not expect more apps in the subnets, for now.

Actual implementation

Code can be find on GitHub, what a plot twist, isn’t it?

As in most of my implementation, I'm using AWS CDK, Typescript, and Projen. Below I will try to show the most crucial part of this implementation, as the whole file has 199 lines.

Constants

let CUSTOM_IMAGE: string = '1dev/server:11.0.9';
let DOMAIN_NAME: string = '3sky.in';

I'm using two hardcoded values (3, if counting the CIDR range). As you can read CUSTOM_IMAGE is an image for our ECS Task, and DOMAIN_NAME is our only prerequisite. I strongly recommend having a domain name on AWS, if you're like me, here you can find tips, about setting it up with a small budget. Of course, the external domain is supported as well, but it requires a bit more work and changes in code.

Security groups

Then we have security groups, which is quite important right? Let's use minimal set of open needed traffic, as it's good practice.

const nlbSecurityGroup = new ec2.SecurityGroup(this, 'ALBSecurityGroup', {
  vpc: vpc,
  description: 'Allow HTTPS traffic to ALB',
  allowAllOutbound: true
});

nlbSecurityGroup.addIngressRule(ec2.Peer.anyIpv4(), ec2.Port.tcp(443), 'Allow HTTPS traffic from anywhere');
nlbSecurityGroup.addIngressRule(ec2.Peer.anyIpv4(), ec2.Port.tcp(22), 'Allow SSH traffic from anywhere');

const privateSecurityGroup = new ec2.SecurityGroup(this, 'PrivateSG', {
  vpc: vpc,
  description: 'Allow access from NLB',
  allowAllOutbound: true
});
privateSecurityGroup.addIngressRule(nlbSecurityGroup, ec2.Port.tcp(6610), 'Allow traffic for HTTP to app');
privateSecurityGroup.addIngressRule(nlbSecurityGroup, ec2.Port.tcp(6611), 'Allow traffic for SSH to app');

const efsSecurityGroup = new ec2.SecurityGroup(this, 'EfsSG', {
  vpc: vpc,
  description: 'Allow access from cluster',
  allowAllOutbound: true
});

efsSecurityGroup.addIngressRule(privateSecurityGroup, ec2.Port.tcp(2049), 'Allow traffic to EFS from ECS cluster');

As you can see, we have 3 SGs. One is attached to NLB, the second for our ECS Cluster, and the last one
for EFS service.

EFS

Now let's configure EFS, as it's a new service. At least on this blog. So we need a few things.

const fileSystem = new efs.FileSystem(this, 'EfsFilesystem', {vpc, securityGroup: efsSecurityGroup});

var accessPoint = new efs.AccessPoint(this, 'VolumeAccessPoint', {
  fileSystem: fileSystem,
  path: '/opt/onedev',
  // app is running as root
  createAcl: {
    ownerGid: '0',
    ownerUid: '0',
    permissions: '755'
  },
  posixUser: {
    uid: '0',
    gid: '0'
  }
});

const volume = {
  name: 'volume',
  efsVolumeConfiguration: {
    authorizationConfig: {
      accessPointId: accessPoint.accessPointId,
      iam: 'ENABLED'
    },
    fileSystemId: fileSystem.fileSystemId,
    transitEncryption: 'ENABLED'
  }
};

const ecsTaskDefinition = new ecs.FargateTaskDefinition(this, 'TaskDefinition', {
  memoryLimitMiB: 2048,
  cpu: 1024,
  runtimePlatform: {
    operatingSystemFamily: ecs.OperatingSystemFamily.LINUX,
    cpuArchitecture: ecs.CpuArchitecture.X86_64
  },
  volumes: [volume]
});

const container = ecsTaskDefinition.addContainer('onedev', {
  image: ecs.ContainerImage.fromRegistry(CUSTOM_IMAGE),
  portMappings: [
    {
      containerPort: 6610,
      protocol: ecs.Protocol.TCP
    }, {
      containerPort: 6611,
      protocol: ecs.Protocol.TCP
    }
  ],
  logging: new ecs.AwsLogDriver(
    {streamPrefix: 'onedev'}
  )
});
container.addMountPoints({readOnly: false, containerPath: '/opt/onedev', sourceVolume: volume.name});

First, a file system is attached to the VPC and security group. Then, we can see the access option. So, basically a way of accessing our storage. Please take a look at the createAcl and posixUser methods, as both of them are responsible for our storage security. However in the case of OneDev, we need to
be a root, so… yes, it could be better. At the end, we have volume which is just a JS object here, as code construction is rather unexpected. Then, we're adding volume to our task definition, not to cluster itself! Lastly, we're adding a container to our task, and attaching mount point to our container. Not such complex, right?

Listeners

In the end, we need to specify listeners and targets. In general, it's rather simple port mapping, however, please take a look at protocols settings. Only the listener on port 443 should use the certificate and TLS as protocol. The rest of the objects should use plain TCP. If we decided to attach a certificate to our listener on port 22, ssh will be unable to connect the service.

const listener443 = nlb.addListener('Listener443', {
  port: 443,
  certificates: [nlbcert],
  protocol: elbv2.Protocol.TLS
});

const listener22 = nlb.addListener('Listener22', {
  port: 22,
  protocol: elbv2.Protocol.TCP
});

listener22.addTargets('ECS-22', {
  port: 6611,
  protocol: elbv2.Protocol.TCP,
  targets: [ecsService]
});

listener443.addTargets('ECS-443', {
  port: 6610,
  protocol: elbv2.Protocol.TCP,
  targets: [ecsService]
});

Local ssh

That allows us to use a very simple SSH config.

Host git.3sky.in
  PreferredAuthentications publickey
  IdentitiesOnly=yes
  IdentityFile ~/.ssh/id_rsa
  User kuba

This part is important. The wrong user or IdentityFile, inhered from git config —global, could mess with your git clone command. So as long as you're not using the same user and SSH key pair, please remember about .ssh/config file.

Summary

Nice, now you can host your code on ECS with limited spending. OneDev seems to be a very solid solution, and because it's one container only, it's very easy to manage. In case of backup it's EFS, so we backup it and restore it without issues.
What about spending? Not sure yet, it will require some real-life testing. Based only on vCPU and RAM usage for one month, the cost could be around 35$.

Information flow - how I capture the notes

Jakub Wołynko — Mon, 26 Aug 2024 11:52:16 +0000

Welcome

This post will be different, than usual. It will be soft, flexible, and with late summer vibes. Please stop coding, deploying, or whatever is important you're doing. That will be the most helpful post you will read today! Let's talk about being old, or more correctly about seniority. What will distinguish you as a person with 10 years of experience in your job with newcomers with 6 months? Probably first things that will come to your mind will be:

coding style
knowing framework or technologies
understanding domain or context(as long as you work for the same company)

And that's it. Which for sure it's true. The thing is that most of those items people are able to master in 2-3 years. Also, when you've been working with technology X, for 3 years, probably you don't remember the tech stack of projects from the beginning of your journal. Unfortunately, that is our(old people's) power - general experience, and specific knowledge. That is why today we will focus on capturing notes and, most importantly using them.

My story with second brain

When I started consulting business (end of 2020), I relied that a lot of my knowledge from previous assignments was gone. Even if I build some hacky Jenkins script, great, slim, and unbelievably readable Ansible playbook, or super SQL query, I can't even look at it. Besides the fact, that it was stored in some corp internal documentation system or git repository. I don't have access to it anymore, and it's not about copyright issues or company-specific things. The general solution could be safely reused, at least in my case. That is why I started looking into the best note-taking app. Which was a wrong idea, and I explain it later. Now, let's talk about available solutions. In order of my story.

Notion

Notion, free for one-person team. Currently, probably the most popular application on the whole internet. You can do everything with it. Build a CRM, take notes, and store rows, as you can do with nocodb. Awesome Swiss-army knife, with a web app,
and native clients (officially Mac and Windows). Unfortunately, in those days no offline mode, and privacy concerns, and everything was stored on Notion's side. Bessie last thing, why did I leave it?

way too many features I don’t need
most of the features aren't platform agnostic, even If I can export to Markdown, without databases it becomes useless
lack of full control over my files

Org-mode

orgmode seems to be a big change after using Notion. However, I fell in love with Emacs, so I wanted to use all the best things. It was easy to set up org-agenda, org-roam, etc. Unfortunately, after a while, I noticed that using a non-standard system(not Markdown) could impact my note-sharing capabilities, as well my work docs were mostly built with Markdown. So, a note system that uses Markdown. Also, I switched to VIM :)

Obsidian

Obsidian, free for personal use, however Sync is paid service.
Obsidian is the holly grail of all second brain builders and home of PKM,
Zettelkasten or many other methods lovers. Personally, during the first time, I went too deep into this solution. I tried many plugins, and added workflow, task and project management, presentations, and diagrams, without the correct directory structure and idea. Also, I was struggling with the setting of the Syncing solution, as git crashed from time to time and iCloud worked fine as long as I was using one machine. So switch. Reasons were:

to a complex system, unstructured and messy as with great power comes great responsibility
issues with syncing

Logseq

logseq fully free and open-source Obsidan-like tool with fewer plugins, however, it also gives you a chance to complex everything a lot. I have been using it for less than a year, however at some point, I noticed that I'm writing longer
forms in Obsidian, and daily notes in Logseq. Why? Due Logseq design. It starts everything as a new point
with -, even if it's a standard Markdown. We’re starting everything at a new point. Issues?

not fully clean Markdown, the text mode isn’t suitable for my needs
not such popular as Obsidian = fewer plugins

Joplin

Joplin open-source tool, with paid Sync service. However, it supports WebDav sync. As a user of Fastmail have a lot lot of storage for it. Those parts work great, links, complexity level, and clear Markdown. Themes, mobile app, tags, everything I needed was there. Unfortunately, again, for short notes, my go-to app becomes memos, for long-form BookStack, seems to be the best solution. Why? Firstly my love for self-hosted solutions boomed, also Joplin even if looks perfect for my use case was some reason hard to describe and did not encourage me to write. Soo.. I back to Obsidian.

Obsidian again

So why Obsidian at the end? The flexibility. This tool can be as simple as I need. My current structure is very clean, with the usage of only 6 plugins:

Calendar
Checklist
Excalidraw
Minimal Theme Settings
QuickAdd
Self-hosted LiveSync

File backup is done via my regular NAS backup task, and between-device sync is done via self-hosted obsidian-livesync.

Honourable mention

Tools used for a short period, but for various reasons do not fit my workflow.

memos - great self-hosted app, but it's a more private social-media platform, than, a note-taking tool. Also, it doesn’t work offline.
anytype - looks like a great Notion alternative, however too complex for me. I need a simple tool.
zettlr - great for long form, but missing some daily use functions.
trilium - not sure, about the project's future.
siyuan - a bit too early stage for me.
Noteplan and Plume - not a Markdown, more Apple notes competitors.

Current setup

After 4 years of trying and experimenting, I created a diagram of tools I'm using in the area of knowledge management.

So how it works you can ask?

First I don't use mobile apps like Obsidian on my phone. I only have Apple Notes for handling ideas and capturing the best quotes from the books I'm reading. Then, day after day, I'm putting it into Obsidian.
Then, I have a pocket notebook, with a special note-taking method. It's for the collection of all ideas, During work, or regular life. It's like a life log.
At the end, I have my Obsidian Vault, for storing everything else, excluding blog posts.
For blog posts, I'm using MarkEdit

What does the process look like?

Then how am I taking the handwriting notes? Based on many years of writing, it's still ugly, and hard to read, however, I'm able to guess most of the things. It looks like that:

Summary

The idea behind this post is a bit selfish. I want to back into this in a year and see how it evolves. Now I'm trying to use as less tools and corporate-owned solutions as possible. My goal is to build vendor and internet-independent workflow as possible. All my notes are stored on local NAS, and synced via live sync. Notebooks are stored in a box near my desk. Why? Because based on my experience, changing structures or formats introduces a lot of mess. Some notes are missing, some are duplicated. The tags do not match. Hopefully, now I will be able to stick with my idea and use it efficiently. What does it mean? Just use them, and read from time to time. At the beginning of my journal, I made a lot of notes. Poor quality, not-completed, disconnected, at the end hard to use. Which is a waste of time, if we can’t reuse our knowledge, or we're unable to find what we need.

Meetings on demand(with Jitsi)

Jakub Wołynko — Fri, 26 Jul 2024 09:15:54 +0000

Welcome

Yesterday I decided to solve another,
probably non-existent problem. So for quite a
long time, I was looking for a solution that would allow me
to schedule meetings with friends, family, and clients
(the custom domain looks very professional). Most of them are paid every month.
Which in my case is not a very cost-effective solution,
if I have one meeting per month, sometimes two. Also let's assume, that
being an AI learning material for Google, or Zoom isn't my dream position.

Suddenly, I realized that I could use Jitsi,
and then self-host it. The problem was that I was looking for an on-demand solution.
So I took the official docker image and rewritten it into ECS, which should just work.

Also, I'd like to apologize to everyone, who is waiting for
part two of setting up Keycloak, or EDA use cases. Someday
I will deliver it, just more time is needed. Additionally, during
the last month, I received a lot of emails about challenges related to
setting up the best open-source SSO solution on ECS. That means, that my article
was needed, and at least few people read it. That's cool!

Jitsi

Ok, so let's start with Jitsi. What is it?

Jitsi Meet is a set of Open Source projects
which empower users to use and deploy video
conferencing platforms with state-of-the-art video quality and features.

As you see it's open-source software on Apache-2.0
license.

Services list

As you can see, the post structure is very similar to latest
one. However today instead of a database we need CloudMap.

ECS(Fargate)
AWS Application Load Balancer
AWS CloudMap
Route53 for domain management

Ok, good. Finally, we will deploy something different right?

Not really, but the use case is super useful!

Implementation

As always it could be simpler, and more elegant, but it does the
job. So as a main point, we have tree files.

❯ tree src
src
├── base.ts
├── jitsi.ts
└── main.ts

Based on my splitting approach, I have a dedicated stack for
more static resources(and cheap), so we can keep them forever(let’s say).
Then we have a file dedicated to our Jitsi ECS stack, at the end
you can see the control file aka src/main.ts with needed variables like account ID,
Region, container version, and your domain name. The content of mentioned file is:

import { App, Tags } from 'aws-cdk-lib';
import { BaseStack } from './base';
import { JitsiStack } from './jitsi';

const devEnv = {
  account: '471112990549',
  region: 'eu-central-1',
};

const DOMAIN_NAME: string = '3sky.in';
const JITSI_IMAGE_VERSION: string = 'stable-9584-1';

const app = new App();
const base = new BaseStack(app, 'jitsi-baseline', {
  env: devEnv,
  stackName: 'jitsi-baseline',
  domainName: DOMAIN_NAME,
});

new JitsiStack(app, 'jitsi-instance',
  {
    env: devEnv,
    stackName: 'jitsi-instnace',
    sg: base.privateSecurityGroup,
    listener: base.listener,
    ecsCluster: base.ecsCluster,
    namespace: base.namespace,
    domainName: DOMAIN_NAME,
    // images version
    jitsiImageVersion: JITSI_IMAGE_VERSION,
  },
);

Tags.of(app).add('description', 'Jitsi Temporary Instance');
app.synth();

Networking

This is up to you, however, I decided to use a separate dedicated VPC with
two AZs - ALB requirement. Additionally, I’m using two types of subnets:
PUBLIC, and PRIVATE_WITH_EGRESS.

Then we have, ALB, Route53 assignment, security groups, ECS Cluster, and
a new interesting object namespace. What do I need this for? Let's dive into
docker-compose for a while.

That's my docker-compose for Actual
app, which IMO is a great app for personal budgeting if you're living in the EU.

name: actual
services:
  actual_server:
    container_name: actual
    image: docker.io/actualbudget/actual-server:24.7.0
    expose:
      - 5006
    volumes:
      - data:/data
    networks:
      - proxy
    restart: unless-stopped

networks:
  proxy:
    name: proxy
    external: true

volumes:
  data:
    external: true
    name: actual_data

As you can see we have a construct called networks, in the above case
it is the default bridge type of network with the name proxy and annotation
external which means that a resource is controlled outside the stack.
Mostly is used to connect services between or inside the stack.

And we have the same in Jitsi's original docker-compose
file.

[...]
        networks:
            meet.jitsi:

# Custom network so all services can communicate using a FQDN
networks:
    meet.jitsi:

Now let's back to the AWS-native landscape. ECS does not support networking
in the same way, as docker. To solve that, issue we need to make a private DNS entry
pointed to the prosody container and resolved it by the xmpp.meet.jitsi value.
Sounds complex, right? Fortunately, there is a service called
AWS Cloud Map.

AWS Cloud Map is a cloud resource discovery service.
With Cloud Map, you can define custom names for your application resources,
and it maintains the updated location of these dynamically changing resources.

How to use it? It's simpler, than you could think!

First define a namespace

const namespace = new servicediscovery.PrivateDnsNamespace(this, 'Namespace', {
  name: 'meet.jitsi',
  vpc
});

Define the cloud option inside your FargateService.

const ecsService = new ecs.FargateService(this, 'EcsService', {
  cluster: cluster,
  taskDefinition: ecsTaskDefinition,
  vpcSubnets: {
    subnetType: ec2.SubnetType.PRIVATE_WITH_EGRESS
  },
  securityGroups: [securityGroup],
  cloudMapOptions: {
    name: 'xmpp',
    container: prosody,
    cloudMapNamespace: namespace,
    dnsRecordType: servicediscovery.DnsRecordType.A
  }
});

Then all your containers, and other services in VPC will be able to call
prosody container by simple xmpp.meet.jitsi FQDN! Which is not always a
good thing, which seems to be a good reason for using dedicated VPC.

Containers

As we have structure, and networking, we need to add containers. In our case it is
very simple. To our FargateTaskDefinition,

const ecsTaskDefinition = new ecs.FargateTaskDefinition(this, 'TaskDefinition', {
  memoryLimitMiB: 8192,
  cpu: 4096,
  runtimePlatform: {
    operatingSystemFamily: ecs.OperatingSystemFamily.LINUX,
    cpuArchitecture: ecs.CpuArchitecture.X86_64
  }
});

we are simply adding four containers web, jicofo, jvb, prosody
just like shown below:

ecsTaskDefinition.addContainer('web', {
  image: ecs.ContainerImage.fromRegistry('quay.io/3sky/jitsi-web:' + jitsiImageVersion),
  environment: {
    TZ: 'Europe/Warsaw',
    PUBLIC_URL: 'https://meet.' + theDomainName
  },
  portMappings: [
    {
      containerPort: 80,
      protocol: ecs.Protocol.TCP
    }
  ],
  logging: new ecs.AwsLogDriver(
    {
      streamPrefix: this.stackName + '-web'
    }
  )
});

Add-ons

If you decided to go through my source code, you could spot two additional
points.

Secrets are simplified

That was done on purpose. Even in public repo, you will see:

let JICOFO_AUTH_PASSWORD: string = '2dAZl8Jkeg5cKT/rbJDZcslGCWt1cA3NnF4QqkFFATY=';
let JVB_AUTH_PASSWORD: string = 'Tph1fJEdU6lFY8aTNPz4EpI5iewQXl+Ot17IeGCmvBs=';

The reason is the ephemeral nature of service and build time. AWS SecretManager
takes more time to generate and after that to destroy. Here, for
my production environment I can just change those strings a bit, and be safe
enough during my 1h call, or even a bit longer according to this
link.

Images are using quay.io

image: ecs.ContainerImage.fromRegistry('quay.io/3sky/jitsi-web:' + jitsiImageVersion),

Due to testing and spanning multiple times containers,
it was very easy to be bothered with the Docker Hub rate limit.
As a lazy person, I just created a bash script to migrate everything
To free, no limited quay.io.

#!/bin/bash

# Define the container version as a variable
CONTAINER_VERSION=$1

# Define an array of images
IMAGES=("web" "prosody" "jvb" "jicofo")

# Check if skopeo is installed
if ! command -v skopeo &> /dev/null
then
    echo "skopeo could not be found, please install it first."
    exit 1
fi

# Function to copy image from Docker Hub to Quay
copy_image() {
    local IMAGE_NAME=$1
    local DOCKERHUB_IMAGE="docker.io/jitsi/$IMAGE_NAME:$CONTAINER_VERSION"
    local QUAY_IMAGE="quay.io/3sky/jitsi-$IMAGE_NAME:$CONTAINER_VERSION"

    # Copy the image using skopeo
    skopeo copy docker://$DOCKERHUB_IMAGE docker://$QUAY_IMAGE

    # Check if the skopeo command was successful
    if [ $? -eq 0 ]; then
        echo "Image $IMAGE_NAME:$CONTAINER_VERSION successfully copied from Docker Hub to Quay."
    else
        echo "Failed to copy image $IMAGE_NAME:$CONTAINER_VERSION from Docker Hub to Quay."
    fi
}

# Iterate over the images and copy each one
for IMAGE in "${IMAGES[@]}"
do
    copy_image $IMAGE
done

Summary

As you can see, I'm hosting my meeting platform with the usage of open-source software, AWS, and a bit of code. The solution is rather cheap, not tested against very big meetings, but stable enough to talk with the wife or friend during summertime.
There is no risk of being recorded, or dependent of
Google account ownership. With that in mind, I'm going
to finish overdue articles... or maybe self-host Ollama
at home and replace ChatGPT. Who knows?

Ah and to be fully open, the source code could be found on
GitHub.

Install Keycloak on ECS(with Aurora Postgresql)

Jakub Wołynko — Fri, 21 Jun 2024 11:06:27 +0000

Welcome

If you have read my latest post about accessing RHEL in the cloud,
you may notice that we’re accessing the cockpit console,
via SSM Session manager port forwarding.
That’s not an ideal solution. I’m not talking in bed,
it’s just not ideal(but cheap). Today I realised that using
Amazon WorkSpaces Secure Browser could be interesting, and fun as well.

Unfortunately, this solution required an Identity Provider,
which can serve us with SAML 2.0. The problem is, that most of the
providers like Okta or Ping,
are enterprise-oriented, and you can’t play with them easily.
Of course, you can request a free trial, but it’s 30 days only,
and there is no single-user account.

That is why I decided to use Keycloak.
Then I just need to decide where to deploy it, in the cloud or in the home lab.
Thankfully we have a mind map.

That is why, today we will install Keycloak on AWS,
and in part two, we will connect it with WorkSpaces
and access the RHEL console over the cockpit, as an enterprise-like user.

Keycloak intro

But hey! What is the Keycloak? In simple words it is
Open Source Identity and Access Management tool.

Also, Keycloak provides user federation, strong authentication,
user management, fine-grained authorization, and more.
You can read more on the GitHub project
page, and star it as well.

Services list

To make it a bit more interesting, I will deploy it with:

ECS(Fargate)
AWS Application Load Balancer
separate database, where today we will play with Serverless Aurora in MySQL mode
Route53 for domain management

Ok, good. Finally, we will deploy something different right?

First steps

At the beginning, I’d recommend building basic projects with networks,
ECS components, database, and ALB. Let’s call it an almost dry test.
Just take a look at the code, and break it into smaller parts.

import * as cdk from 'aws-cdk-lib';
import { Construct } from 'constructs';
import * as ec2 from 'aws-cdk-lib/aws-ec2';
import * as rds from 'aws-cdk-lib/aws-rds';
import * as ecs from 'aws-cdk-lib/aws-ecs';
import * as elbv2 from 'aws-cdk-lib/aws-elasticloadbalancingv2'

On top, as you can see, I have imports. It’s worth to mention,
that AWS Aurora Serverless V2 is using aws-rds package, and modern load balancers,
like ALB is included in aws-elasticloadbalancingv2, not standard aws-elasticloadbalancing.

Firstly we have a network. Today we will have regular 3-tier architecture,
with a really small mask - /28, which means 11 IPs; regular 16 - 5 (AWS specific).
Besides of that two AZs for Aurora HA, so 2 NAT Gateways as well.

const vpc = new ec2.Vpc(this, 'VPC', {
  ipAddresses: ec2.IpAddresses.cidr("10.192.0.0/20"),
  maxAzs: 2,
  enableDnsHostnames: true,
  enableDnsSupport: true,
  restrictDefaultSecurityGroup: true,
  subnetConfiguration: [
    {
      cidrMask: 28,
      name: "public",
      subnetType: ec2.SubnetType.PUBLIC
    }, {
      cidrMask: 28,
      name: "private",
      subnetType: ec2.SubnetType.PRIVATE_WITH_EGRESS
    }, {
      cidrMask: 28,
      name: "database",
      subnetType: ec2.SubnetType.PRIVATE_ISOLATED
    }
  ]
});

Next we very simple setup of the application load balancer.
It’s listed on port 80(no TLS), it’s internet facing, and exists in our VPC,
inside subnets with type “PUBLIC”.

const alb = new elbv2.ApplicationLoadBalancer(this, 'ApplicationLB', {
      vpc: vpc,
      vpcSubnets: {
        subnetType: ec2.SubnetType.PUBLIC,
      },
      internetFacing: true
    });

    const listener = alb.addListener('Listener', {
      port: 80,
      open: true,
    });
});

Here the same, we have a very basic setup of Postgres Aurora, without security groups
or deletion protection.

However, we’re setting it in the correct set of subnets,
and what is more important we’re checking our Postgres engine version,
as the latest is not always supported. If you would like to check it
on our own, here is the command, which shows you available engines.

aws rds describe-db-engine-versions \
    --engine aurora-mysql \
    --filters Name=engine-mode,Values=serverless

BUT, if you would like to use Aurora Serverless V2, you need to use rds.DatabaseCluster,
so our code will be, and as you can see we can use a higher version of Postgres:

new rds.DatabaseCluster(this, 'AuroraCluster', {
  engine: rds.DatabaseClusterEngine.auroraMysql(
    {version: rds.AuroraMysqlEngineVersion.VER_2_11_4}
  ),
  vpc: vpc,
  credentials: {
    username: 'keycloak',
    // WARNING: This is wrong, do not work this way
    password: cdk.SecretValue.unsafePlainText('password')
  },
  // NOTE: use this rather for testing
  deletionProtection: false,
  securityGroups: [auroraSecurityGroup],
  vpcSubnets: {
    subnetType: ec2.SubnetType.PRIVATE_ISOLATED
  },
  writer: rds.ClusterInstance.serverlessV2('ClusterInstance', {scaleWithWriter: true})
});

Great, now we can configure:

ECS Cluster with Fargate Capacity Providers
Fargate Task Definition
container
ECS Service
ALB TargetGroup registration

and check if we will be able to
access our container from the regular Internet.

const ecsCluster = new ecs.Cluster(this, 'EcsCluster', {
  clusterName: 'keycloak-ecs-cluster',
  // NOTE: add some logging
  containerInsights: true,
  enableFargateCapacityProviders: true,
  vpc: vpc
})
const ecsTaskDefinition = new ecs.FargateTaskDefinition(
  this,
  'TaskDefinition',
  {
  memoryLimitMiB: 512,
  cpu: 256,
  runtimePlatform: {
    operatingSystemFamily: ecs.OperatingSystemFamily.LINUX,
    cpuArchitecture: ecs.CpuArchitecture.X86_64
  }
});
const container = ecsTaskDefinition.addContainer('keycloak', {
  image: ecs.ContainerImage.fromRegistry('nginx'),
  portMappings: [
    {
      containerPort: 80,
      protocol: ecs.Protocol.TCP
    }
  ]
});
const ecsService = new ecs.FargateService(this, 'EcsService', {
  cluster: ecsCluster,
  taskDefinition: ecsTaskDefinition,
  vpcSubnets: {
    subnetType: ec2.SubnetType.PRIVATE_WITH_EGRESS
  }
});
ecsService.registerLoadBalancerTargets({
  containerName: 'keycloak',
  containerPort: 80,
  newTargetGroupId: 'ECS',
  listener: ecs.ListenerConfig.applicationListener(
    listener,
    { protocol: elbv2.ApplicationProtocol.HTTP })
},);

As you see the logic here is straightforward.
First, we need a cluster, with enabled FargateCapacityProvider,
and we would like to place it in our VPC.
Then we have a task, that needs to be an instance of class ecs.FargateTaskDefinition.
Next, we’re adding a container object with a Nginx docker image and port mapping.
Additionally, we need a service that combines all elements.
At the end, we register our service, as Load Balancer Target.

This configuration should allow us to validate our application skeleton,
and make sure that basic components are up, and we don’t mess something.
That is how I like to work, make a simple landscape, then tweak it according to the needs.

Keycloak configuration

Now, we need to clarify a few things.
First, as we’re planning to run Keycloak in container, that is why we need
to go through this doc,
then as we’re using Aurora, we need this.

Second, there is no support for IAM Roles, which is why we need to use a username/password.
However, if you feel stronger than me with AWS SDK for Java,
and have some time, I’m sure that the Keycloak team will be more than happy to review your PR.

So after reading the documentation, looks like I need to add the JDBC driver to the image,
which requires building a custom image. It will be very simple
but require some project modification. Additionally, we need some environment variables:

const container = ecsTaskDefinition.addContainer('keycloak', {
  image: ecs.ContainerImage.fromRegistry('quay.io/keycloak/keycloak:24.0'),
  environment: {
    KEYCLOAK_ADMIN: "admin",
    KEYCLOAK_ADMIN_PASSWORD: "admin",
    KC_DB: "postgres",
    KC_DB_USERNAME: "keycloak",
    KC_DB_PASSWORD: "password",
    KC_DB_SCHEMA: "public",
    KC_LOG_LEVEL: "INFO, org.infinispan: DEBUG, org.jgroups: DEBUG"
  },
  portMappings: [
    {
      containerPort: 80,
      protocol: ecs.Protocol.TCP
    }
  ]
});

Stop. Why it doesn’t work?

Look good right? No, no, and no! Making this project work
forced me to spend much more time than I expected. Why? That’s simple,
Keycloak is complex software, which is why a lot of configuration is
needed. However, let’s start from the beginning.

Building custom docker image

I decided, that a public docker image from Aurora support would be great,
and maybe helpful for someone. That is why you can find it on quay.
Also whole project repo is public, and accessible here.
Besides GitHub Actions, Dockerfile is the most important part, so here we go:

ARG VERSION=25.0.1
ARG BUILD_DATE=today

FROM quay.io/keycloak/keycloak:${VERSION} as builder

LABEL vendor="3sky.dev" \
    maintainer="Kuba Wolynko <kuba@3sky.dev>" \
    name="Keyclock for Aurora usage" \
    arch="x86" \
    build-date=${BUILD_DATE}

# Enable health and metrics support
ENV KC_HEALTH_ENABLED=true
ENV KC_METRICS_ENABLED=true
ENV KC_DB=postgres
ENV KC_DB_DRIVER=software.amazon.jdbc.Driver

WORKDIR /opt/keycloak
# use ALB on top, self-sign is fine here
RUN keytool -genkeypair \
    -storepass password \
    -storetype PKCS12 \
    -keyalg RSA \
    -keysize 2048 \
    -dname "CN=server" \
    -alias server \
    -ext "SAN:c=DNS:localhost,IP:127.0.0.1" \
    -keystore conf/server.keystore

ADD --chmod=0666 https://github.com/awslabs/aws-advanced-jdbc-wrapper/releases/download/2.3.6/aws-advanced-jdbc-wrapper-2.3.6.jar /opt/keycloak/providers/aws-advanced-jdbc-wrapper.jar
RUN /opt/keycloak/bin/kc.sh build

FROM quay.io/keycloak/keycloak:${VERSION}
COPY --from=builder /opt/keycloak/ /opt/keycloak/

ENTRYPOINT ["/opt/keycloak/bin/kc.sh"]

Worth mentioning is the fact, that the build command needs to be
executed after getting aws-advanced-jdbc-wrapper,
or enabling health checks. Also, I’m using a self-sign certificate
as AWS will provide public cert on the ALB level.

Documentation

When you are going through official Keycloak documentation
you will be able to read something like this:

I’m not a native speaker, so for me, it means:
Put aws-wrapper into your JDBC URL. But, it’s not true.
It means that the result will be jdbc:aws-wrapper:postgresql://,
which is not what the application can consume!
To work with Aurora Postgresql, our connection string must be:

Edit: I was wrong. aws_wrapper needs to be included in JDBC URL,
to use advanced wrapper's functions. However environment variable KC_DB_DRIVER=software.amazon.jdbc.Driver,
need to be present in Dockerfile before running /opt/keycloak/bin/kc.sh build command.

KC_DB_URL: 'jdbc:aws-wrapper:postgresql://' + theAurora.clusterEndpoint.hostname + ':5432/keycloak',

Note: keycloak at the end is the database name, and can be customized.

Believe it or not, this misunderstanding costs me around the week of debugging...

Cookie not found

When after a while I was able to start an app and type admin/admin,
I was welcomed with this error message:

Cookie not found. Please make sure cookies are enabled in your browser.

After some investigation, and reading stackoverflow threads,
the decision was to add TLS and see what would happen. To do that in rather a simple
manner, a hosted zone with a public registered domain was needed. And you can read about it
setting it up here. In case of adding it to project we can simply write:

const zone = route53.HostedZone.fromLookup(this, 'Zone', {domainName: DOMAIN_NAME});
new route53.CnameRecord(this, 'cnameForAlb', {
  recordName: 'sso',
  zone: zone,
  domainName: alb.loadBalancerDnsName,
  ttl: cdk.Duration.minutes(1)
});
const albcert = new acme.Certificate(this, 'Certificate', {
  domainName: 'sso.' + DOMAIN_NAME,
  certificateName: 'Testing keyclock service', // Optionally provide an certificate name
  validation: acme.CertificateValidation.fromDns(zone)
});
this.listener = alb.addListener('Listener', {
  port: 443,
  open: true,
  certificates: [albcert]
});

Health check

This one is tricky. The target Group attached to the Application Load Balancer requires healthy targets.
Using the native ECS method as described above does not meet our needs, as we expected:

dedicated path /health
different port(9000), than main container port(8433)
if we decide to stick with / path, we need to accept port 302

As you see we need to change a method to a more traditional one:

theListner.addTargets('ECS', {
  port: 8443,
  targets: [ecsService],
  healthCheck: {
    port: '9000',
    path: '/health',
    interval: cdk.Duration.seconds(30),
    timeout: cdk.Duration.seconds(5),
    healthyHttpCodes: '200'
  }
});

NOTE: default HC port could be changed, but it adds additional complexity.

Build time

Playing with ECS and database in the same CDK stack is not the best possible idea. Why?
Let's imagine the situation when your health checks are failing.
Deployment is in progress and still rolling. During this period,
you can't change your stack. But you can delete it…, and recreate it.
Even if the networking part is fast, spanning Aurora up and down,
could add to our change time by around 12 minutes.
That's not a bed, but still, it's quite easy to avoid it.
The change was simple. I created a new stack, dedicated to ECS,
and split the app into two parts:

infra (network and database)
ECS (container and services)

This simple action shows that the modularity of Infrastructure as Code could be more
important than we're usually thinking.

Final declaration

Slowly going to the end, the ECS dedicated part will look like that:

ecsTaskDefinition.addContainer('keycloak', {
  image: ecs.ContainerImage.fromRegistry(CUSTOM_IMAGE),
  environment: {
    KEYCLOAK_ADMIN: 'admin',
    KEYCLOAK_ADMIN_PASSWORD: 'admin',
    KC_DB: 'postgres',
    KC_DB_URL: 'jdbc:postgresql://' + theAurora.clusterEndpoint.hostname + ':5432/keycloak',
    KC_DB_USERNAME: 'keycloak',
    KC_DB_PASSWORD: theSecret.secretValueFromJson('password').toString(),
    KC_HOSTNAME_STRICT: 'false',
    KC_HTTP_ENABLED: 'false',
    KC_DB_DRIVER: 'software.amazon.jdbc.Driver',
    KC_HEALTH_ENABLED: 'true'
  },
  portMappings: [
    {
      containerPort: 8443,
      protocol: ecs.Protocol.TCP
    }, {
      containerPort: 9000,
      protocol: ecs.Protocol.TCP
    }
  ],
  logging: new ecs.AwsLogDriver(
    {streamPrefix: 'keycloak'}
  ),
  command: ['start', '--optimized']
});

Final architecture

As we all love images, that's the final digram of our setup:

Summary

That's the end of setting keycloak on ECS with the usage of Aurora Postgres.
I didn't see it anywhere on the Internet, so for at least a moment, that is
the only working example and blog post available publicly.
Additionally, a solution was tested with versions 24.0 and the latest 25.0.1 of
Keycloak base images. What will be next? As I mentioned at the beginning.
Setting up Worksapces with my own SSO solution, however, firstly we need to
configure Keyclock, which I’m almost sure, will be fun as well.
Ah and to be fully open, source code could be found on GitHub.

Red Hat Enterprise Linux on AWS Cloud

Jakub Wołynko — Thu, 23 May 2024 11:26:23 +0000

Welcome

Let’s talk about basic IT operations included in the everyday tasks range. For example, accessing VMs. As you may realize (or not) - not everyone is using immutable infrastructure. Especially when their core business isn’t IT, and they are a bit bigger than 2-pizza team. That is why today we will talk about accessing the console of Red Hat Enterprise Linux 9.3 in AWS Cloud. I will show you the 3 most useful methods - in my opinion; there are no statistics.

Initial note

Due to the fact, that I appreciate AWS CDK, all examples today will be written in TypeScript.
Additionally, I decided to use shared stacks and store VPC config separately,
as well as default instance properties.
The network is very simple, one public subnet,
one private subnet, and one NAT(default option when using subnetType: ec2.SubnetType.PRIVATE_WITH_EGRESS).

import * as cdk from 'aws-cdk-lib';
import * as ec2 from 'aws-cdk-lib/aws-ec2';

export class SharedNetworkStack extends cdk.Stack {
  public readonly vpc: ec2.Vpc;
  constructor(scope: cdk.App, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    cdk.Tags.of(this).add("description", "Shared Network");
    cdk.Tags.of(this).add("organization", "3sky.dev");
    cdk.Tags.of(this).add("owner", "3sky");

    this.vpc = new ec2.Vpc(this, 'TheVPC', {
      ipAddresses: ec2.IpAddresses.cidr("10.192.0.0/20"),
      maxAzs: 1,
      enableDnsHostnames: true,
      enableDnsSupport: true,
      restrictDefaultSecurityGroup: true,
      subnetConfiguration: [
        {
          cidrMask: 28,
          name: "public",
          subnetType: ec2.SubnetType.PUBLIC,
        },
        {
          cidrMask: 28,
          name: "private",
          subnetType: ec2.SubnetType.PRIVATE_WITH_EGRESS,
        },
      ],
    });
  }
}

As a base AMI for all instances, I will be using the publicly available latest RH build: amazon/RHEL-9.3.0_HVM-20240117-x86_64-49-Hourly2-GP3. With the smallest possible instance size. Everything is sorted in file called bin/rhel.ts

#!/usr/bin/env node
import 'source-map-support/register';
import * as cdk from 'aws-cdk-lib';
import * as ec2 from 'aws-cdk-lib/aws-ec2';
import { SSHStack } from '../lib/ssh-stack';
import { ICStack } from '../lib/ic-stack';
import { SSMStack } from '../lib/ssm-stack';
import { SharedNetworkStack } from '../lib/shared-network';

const app = new cdk.App();

const network = new SharedNetworkStack(app, 'SharedNetworkStack');

const defaultInstanceProps = {
    vpc: network.vpc,
    machineImage: ec2.MachineImage.genericLinux({
        // amazon/RHEL-9.3.0_HVM-20240117-x86_64-49-Hourly2-GP3
        "eu-central-1": "ami-0134dde2b68fe1b07",
    }),
    instanceType: ec2.InstanceType.of(
        ec2.InstanceClass.BURSTABLE2,
        ec2.InstanceSize.MICRO,
    ),
};

new SSHStack(app, 'SSHStack', {
    instanceProps: defaultInstanceProps,
    vpc: network.vpc,
});

new ICStack(app, 'ICStack', {
    instanceProps: defaultInstanceProps,
    vpc: network.vpc,
});

new SSMStack(app, 'SSMStack', {
    instanceProps: defaultInstanceProps,
});

SSH

Let’s start with the basics. Regular SSH, what do we need to make this possible?

SSH Key Pair
ssh-server and ssh-client installed
connection to configured port(default: 22)
Bastion Host, as we’re simulating enterprise

setup

The initial assumption is, that we already have a key pair if not, please generate it with the following command:

ssh-keygen \
    -t ed25519 \
    -C "aws@local-testing" \
    -f ~/.ssh/id_ed25519_local_testing

Let’s back to code, we’re using a much too open Security Group for the Bastion host, regular in-VPC SG, and two hosts with the same ssh-key configured. That is why the created stack definition is rather simple:

import * as cdk from 'aws-cdk-lib';
import { Construct } from 'constructs';
import * as ec2 from 'aws-cdk-lib/aws-ec2';

export interface SSHStackProps extends cdk.StackProps {
  vpc: ec2.Vpc;
  instanceProps: any;
}
export class SSHStack extends cdk.Stack {
  constructor(scope: Construct, id: string, props: SSHStackProps) {
    super(scope, id, props);

    const theVPC = props.vpc;
    const theProps = props.instanceProps;

    // WARNING: change key material to your own
    const awsKeyPair = new ec2.CfnKeyPair(this, "localkeypair", {
      publicKeyMaterial:
        "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIJxYZEBNRLXmuign6ZgNbmaSK7cnQAgFpx8cCscoqVed local",
      keyName: "localawesomekey",
    });

    const myKeyPair = ec2.KeyPair.fromKeyPairAttributes(this, "mykey", {
      keyPairName: awsKeyPair.keyName,
    });

    const tooopenSG = new ec2.SecurityGroup(this, "tooopenSG", {
      securityGroupName: "Allow all SSH traffic",
      vpc: theVPC,
      allowAllOutbound: false
    });

    tooopenSG.addIngressRule(
      // NOTE: we should use a more specific network range with
      // ec2.Peer.ipv4("x.x.x.x/24")
      ec2.Peer.anyIpv4(),
      ec2.Port.tcp(22),
      "Allow SSH",
      false,
    );

    const defaultSG = new ec2.SecurityGroup(this, "regularSG", {
      securityGroupName: "Regular in-VPC SG",
      vpc: theVPC,
      allowAllOutbound: false
    });

    defaultSG.addIngressRule(
      ec2.Peer.ipv4(theVPC.vpcCidrBlock),
      ec2.Port.tcp(22),
      "Allow SSH inside VPC only"
    );


    const bastion = new ec2.Instance(this, 'bastion-host', {
      instanceName: 'bastion-host',
      vpcSubnets: {
        subnetType: ec2.SubnetType.PUBLIC,
      },
      securityGroup: tooopenSG,
      keyPair: myKeyPair,
      ...theProps,
    });

    const instance = new ec2.Instance(this, 'host', {
      instanceName: 'host',
      vpcSubnets: {
        subnetType: ec2.SubnetType.PRIVATE_WITH_EGRESS,
      },
      securityGroup: defaultSG,
      keyPair: myKeyPair,
      ...theProps,
    });

    new cdk.CfnOutput(this, "bastionIP", {
      value: bastion.instancePublicIp,
      description: "Public IP address of the bastion host",
    });

    new cdk.CfnOutput(this, "instnaceIP", {
      value: instance.instancePrivateIp,
      description: "Private IP address of thsh host",
    });
  }
}

Then after executing cdk deploy SSHStack, and waiting around 200s, we should be able to see:


 ✅  SSHStack

✨  Deployment time: 200.17s

Outputs:
SSHStack.bastionIP = 3.121.228.159
SSHStack.instnaceIP = 10.192.0.24

Great, now we can use our ssh key and ec2-user, for connection with our instance.

$ ssh ec2-user@3.121.228.159 -i ~/.ssh/id_ed25519_local

Register this system with Red Hat Insights: insights-client --register
Create an account or view all your systems at https://red.ht/insights-dashboard
[ec2-user@ip-10-192-0-6 ~]$ clear
[ec2-user@ip-10-192-0-6 ~]$
logout
Connection to 3.121.228.159 closed.

Ok, for accessing our regular target host I strongly recommend using a regular ~/.ssh/config file, with the following content:

Host aws-bastion
  PreferredAuthentications publickey
  IdentitiesOnly=yes
  IdentityFile ~/.ssh/id_ed25519_local
  User ec2-user
  Hostname 3.76.116.53

Host aws-host
  PreferredAuthentications publickey
  IdentitiesOnly=yes
  ProxyJump jump
  IdentityFile ~/.ssh/id_ed25519_local
  User ec2-user
  Hostname 10.192.0.24

The good thing about it is that it’s very easy to configure Ansible with it. Our inventory file will be just simple:

[bastion]
aws-bastion

[instance]
aws-host

[aws:children]
aws-bastion
aws-host

However, in the case of a real-world system, I would recommend using Ansible dynamic inventory, based on proper tagging.

props

easy-to-implement solution
standard SSH, so we can use Ansible just after deployment
no additional costs(besides Bastion host)

cons

exposing VMs to the public internet isn’t the most secure solution
we need to manage SSH key pair
we need to manage users
we need to manage accesses manually, from the OS level
no dedicated logging solution, besides Syslog

SSM Session Manager

Setting SSM based on documentation could be a bit more challenging, as we need to:

install SSM agent
configure role and instance profile

setup

import * as cdk from 'aws-cdk-lib';
import { Construct } from 'constructs';
import * as ec2 from 'aws-cdk-lib/aws-ec2';
import * as iam from 'aws-cdk-lib/aws-iam'

export interface SSMStackProps extends cdk.StackProps {
  instanceProps: any;
}
export class SSMStack extends cdk.Stack {
  constructor(scope: Construct, id: string, props: SSMStackProps) {
    super(scope, id, props);

    const theProps = props.instanceProps;

    const ssmRole = new iam.Role(this, "SSMRole", {
      assumedBy: new iam.ServicePrincipal("ec2.amazonaws.com"),
      managedPolicies: [
        iam.ManagedPolicy.fromAwsManagedPolicyName("AmazonSSMManagedInstanceCore")
      ],
      roleName: "SSMRole"
    });
    new iam.InstanceProfile(this, "SSMInstanceProfile", {
      role: ssmRole,
      instanceProfileName: "SSMInstanceProfile"
    });

    const userData = ec2.UserData.forLinux();
    userData.addCommands(
      'set -o xtrace',
      'sudo dnf install -y https://s3.amazonaws.com/ec2-downloads-windows/SSMAgent/latest/linux_amd64/amazon-ssm-agent.rpm',
      'sudo systemctl enable amazon-ssm-agent',
      'sudo systemctl start amazon-ssm-agent'
    );

    const instnace = new ec2.Instance(this, 'instance-with-ssm', {
      instanceName: 'instance-with-ssm',
      vpcSubnets: {
        subnetType: ec2.SubnetType.PRIVATE_WITH_EGRESS,
      },
      role: ssmRole,
      allowAllOutbound: true,
      detailedMonitoring: true,
      userData: userData,
      ...theProps,
    });


    new cdk.CfnOutput(this, "HostID", {
      value: instnace.instanceId,
      description: "ID of the regular host",
    });

    new cdk.CfnOutput(this, "hostDNS", {
      value: instnace.instancePrivateDnsName,
      description: "Hostname of the regular host",
    });
  }
}

As you can see, we need to specify the role and instance profile(which can’t be displayed in GUI), and then we place the instance in a private subnet with specific user data and role. After a while we should be able to connect via CLI:

$ aws ssm start-session --target i-0110d03f4713d475c


Starting session with SessionId: kuba@3sky.dev-6n5goh43iisz45cmvpht54ize4
sh-5.1$ sudo su
[root@ip-10-192-0-26 bin]#

Or via GUI:

bonus: cockpit

As we’re using the Red Hat system(however also available on Ubuntu etc), and SSM supports port forwarding we can utilize the power of the cockpit. For example, manage subscriptions, check security recommendations, and connect with Red Hat Insights. How to do it? Login to our instance, create an admin user with a password, and then start the port forwarding session.

# login to host
$ aws ssm start-session --target i-0113d03f4713d412b

# become a root and create the needed user
sh-5.1$ sudo su
[root@ip-10-192-0-24:~]$ sudo useradd kuba
[root@ip-10-192-0-24:~]$ sudo passwd kuba
[root@ip-10-192-0-24:~]$ sudo usermod -aG wheel kuba
[root@ip-10-192-0-24:~]$ exit
sh-5.1$ exit

## start port-forwarding from the workstation
$ aws ssm start-session \
  --target i-0113d03f4713d412b 
  --document-name AWS-StartPortForwardingSessionToRemoteHost \
  --parameters '{"portNumber":["9090"],"localPortNumber":["9090"],"host":["ip-10-192-0-24"]}'

props

straightforward setup
allow the user to access the instance from GUI and CLI
does not require using or storing ssh-keys
has built in monitoring with CloudWatch
provides the ability to restrict access based on IAM
support port forwarding (useful for DB access)

cons

using Ansible will be challenging(however supported)
requires more AWS-specific knowledge than Bastion host
in case of failure, push us to annoying debugging

EC2 instance connect

Our EC2 Instance Connect setup and configuration will be based on official documentation.
Here are the main prerequisites we have:

installed ec2-instance-connect packages on our host, which are not included in the default Red Hat build
EC2 instance connect endpoint placed in the corresponding network
open network connection to instance security group on TCP/22
open network connection from EC2 instance to connect security group to instances on TCP/22

setup

Based on these prerequisites content of our file is as:

import * as cdk from 'aws-cdk-lib';
import { Construct } from 'constructs';
import * as ec2 from 'aws-cdk-lib/aws-ec2';

export interface ICStackProps extends cdk.StackProps {
  vpc: ec2.Vpc;
  instanceProps: any;
}
export class ICStack extends cdk.Stack {
  constructor(scope: Construct, id: string, props: ICStackProps) {
    super(scope, id, props);

    const theVPC = props.vpc;
    const theProps = props.instanceProps;

    const iceSG = new ec2.SecurityGroup(this, "iceSG", {
      securityGroupName: "Instance Connect SG",
      vpc: theVPC,
      allowAllOutbound: false
    });

    iceSG.addEgressRule(
      ec2.Peer.ipv4(theVPC.vpcCidrBlock),
      ec2.Port.tcp(22),
      "Allow outbound traffic from SG",
    );

    // WARNING: We need outbound for package installation
    const iceSGtoVM = new ec2.SecurityGroup(this, "iceSGtoVM", {
      securityGroupName: "Allow access over instance connect",
      vpc: theVPC,
    });

    iceSGtoVM.addIngressRule(
      iceSG,
      ec2.Port.tcp(22),
      "Allow SSH traffic from iceSG",
    );

    new ec2.CfnInstanceConnectEndpoint(this, "myInstanceConnectEndpoint", {
      securityGroupIds: [iceSG.securityGroupId],
      subnetId: theVPC.privateSubnets[0].subnetId
    });

    const userData = ec2.UserData.forLinux();
    userData.addCommands(
      'set -o xtrace',
      'mkdir /tmp/ec2-instance-connect',
      'curl https://amazon-ec2-instance-connect-us-west-2.s3.us-west-2.amazonaws.com/latest/linux_amd64/ec2-instance-connect.rpm -o /tmp/ec2-instance-connect/ec2-instance-connect.rpm',
      'curl https://amazon-ec2-instance-connect-us-west-2.s3.us-west-2.amazonaws.com/latest/linux_amd64/ec2-instance-connect-selinux.noarch.rpm -o /tmp/ec2-instance-connect/ec2-instance-connect-selinux.rpm',
      'sudo yum install -y /tmp/ec2-instance-connect/ec2-instance-connect.rpm /tmp/ec2-instance-connect/ec2-instance-connect-selinux.rpm'
    );

    const instnace = new ec2.Instance(this, 'instance-with-ic', {
      instanceName: 'instance-with-ic',
      vpcSubnets: {
        subnetType: ec2.SubnetType.PRIVATE_WITH_EGRESS,
      },
      securityGroup: iceSGtoVM,
      allowAllOutbound: true,
      detailedMonitoring: true,
      userData: userData,
      ...theProps,
    });

    new cdk.CfnOutput(this, "HostIP", {
      value: instnace.instanceId,
      description: "Public IP address of the regular host",
    });
  }
}

As you can see, the longest part is just user-data for downloading and installing needed packages. What is important in my opinion, we always should test it before deploying it on production. In case of errors with the installation process, debugging will be hard and will require adding a bastion host with keys, and instance recreation.

After deploying a stack(a bit longer this time), we should be able to access our instance with CLI:

$ aws ec2-instance-connect ssh --instance-id i-08385338c2614df28

The authenticity of host '10.192.0.26 (<no hostip for proxy command>)' can't be established.
ED25519 key fingerprint is SHA256:BAxtwbZYKsK6hTJbvqOGgulGYftNQHZHMSpBkIGRTeY.
This key is not known by any other names.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '10.192.0.26' (ED25519) to the list of known hosts.
Register this system with Red Hat Insights: insights-client --register
Create an account or view all your systems at https://red.ht/insights-dashboard
[ec2-user@ip-10-192-0-26 ~]$

or via GUI:

props

straightforward setup
allow the user to access the instance from GUI and CLI
does not require using or storing ssh-keys
has built in monitoring with CloudWatch
provides the ability to restrict access based on IAM

cons

using Ansible will be very challenging(no official support or plugin)
requires more AWS-specific knowledge than Bastion host
in case of failure, push us to annoying debugging

Final notes

network setup was simplified; one AZ; 2 subnets
security groups were to open, especially outbound rules(due to package downloading)
SSM could be used over endpoints as well:
- com.amazonaws.:aws-region:.ssm
- com.amazonaws.:aws-region:.ssmmessages
- com.amazonaws.:aws-region:.ssmmessages
we should use already pre-built AMI to avoid this issue
repo as always can be find here

Summary

As you can see setting up RHEL in AWS isn’t such hard, for sure it’s more expensive and requires ec2-connect-instance or SSM agent installation(Amazon Linux does not), but if we’re going to use RHEL in the cloud, probably we have good reason to do so. For example, great support, Insights, or Cloud Console, but my job was to show possibilities and do it with style.