DEV Community: Misha Bragin

AWS Lambda Serverless Security. Mistakes, Oversights, and Potential Vulnerabilities

Misha Bragin — Fri, 03 May 2024 09:07:15 +0000

Amazon Web Services (AWS) Lambda is a serverless function-as-a-service (FaaS) platform that lets you deploy, run, and scale code in the cloud as self-contained functions without having to manually configure any infrastructure. Lambda runs your functions on demand in response to specific events, such as an HTTP request from the internet or activity in another AWS service.

Using Lambda helps accelerate your development cycles by letting engineers stay focused on code, but going serverless is also a potential security risk. Incorrectly configured functions can harbor several kinds of vulnerabilities that can be exploited to allow unauthorized access or cause instability in your deployments.

In this guide, you'll explore some common Lambda security pitfalls and learn how to properly protect your functions. You need to understand how serverless changes the cloud security model so that you can confidently adopt the architecture while keeping your apps, data, and users safe.

Why Does AWS Lambda Serverless Security Matter

Security should be a primary concern when deploying a new AWS Lambda serverless function. Although Lambda handles infrastructure management for you, this doesn't mean you can trust its default configuration to automatically secure your functions and related resources.

Lambda operates under a shared responsibility security model. AWS secures the physical infrastructure that runs the service—including software components, such as the operating system and Lambda platform—but you're responsible for protecting your functions, their code, the generated data, and network routes to other stack components.

Correctly configuring serverless security is vital so you can safely use functions in business-critical scenarios, such as when your code processes sensitive or proprietary data. There are two main aspects to consider:

Code security: This relates to vulnerabilities that exist within your function's code. It includes areas such as dependency supply chain security and the use of effective user input validation and sanitization.
Function security: This covers the security of your Lambda functions and AWS environment, including the correct configuration of rate-limiting controls, resource constraints, and authorization requirements.

Unfortunately, it's easy to make errors in both of these areas.

Common AWS Lambda Security Mistakes and Oversights

Now that you've seen why it's important to secure your Lambda functions, let's analyze some of the most commonly experienced mistakes and oversights. Your functions could be vulnerable to these threats if you haven't already taken action to address them.

Inadequate Function-Level Permission Restrictions

Inadequate permission checks within your function's code can allow attackers to gain unauthorized access to your data. This risk can be present in any service, but the fast-paced Lambda development cycle can mean the threat easily goes unnoticed. Developers may also have a false sense of security that resources running within AWS or Lambda are already being protected.

You can address this problem by auditing all code for security issues before deployment. Additionally, you should shift security left and identify any privileged actions before you start development. This ensures that engineers know how to implement relevant permission checks as they build their functions.

Insufficient Error Handling and/or Logging

Functions should be observable so that you can spot errors and analyze suspected attack activity. Codes with insufficient error reporting and logging mask security incidents, making it easier for threat actors to acquire long-term access.

Log collection issues can be easily solved by instrumenting your function to emit logs on its standard output and error streams. Lambda automatically sends this output to the AWS CloudWatch Logs management service, allowing you to monitor log output and verify your function is working as expected.

Insufficient Resource Allocation

Lambda functions that have been allocated insufficient memory or CPU resources can cause resource exhaustion. At best, this leads to instability; at worst, malicious users could exploit it to achieve a denial-of-service (DoS) attack against your functions.

Resource constraints are controlled by setting the memory available to your Lambda functions. The available CPU capacity is then scaled automatically based on the amount of memory you've requested. You should assess your function's memory requirement before you deploy, then select an allocation that provides enough headroom for real-world usage. It's also important to set up CloudWatch alerts to receive notifications when a function invocation approaches its memory limit or exceeds defined thresholds for CPU or IO activity.

Excessive Resource Allocation

It's not just inadequate resource allocation that poses a security threat—assigning too much memory can be equally impactful. Because Lambda pricing is determined based on the amount of memory allocated and the total duration of each invocation, excessive allocation causes an unnecessary increase in your operating costs. Attackers could abuse this by directing malicious traffic to your services, forcing you to accept a higher bill.

To mitigate this threat, ensure you regularly monitor the actual resource utilization of your functions using the metrics available in CloudWatch. You can then rightsize your memory allocations to achieve an optimal balance of performance and cost efficiency.

Use of Insecure Dependencies

Functions that depend on vulnerable or outdated third-party packages pose security risks when running in Lambda, just like when any other deployment method is used.

Using unmaintained packages exposes you to actively exploited Common Vulnerabilities and Exposures (CVEs), insecure or buggy code, and the prospect of backdoors or intentional weaknesses inserted by malicious authors. You should practice safe supply chain security to identify these problems before you deploy, such as by generating a software bill of materials (SBOM) that lists your dependencies and any known issues they contain.

Data Injection

Unprotected Lambda functions can be vulnerable to data injection issues, such as improper user input sanitization or acceptance of malformed commands generated by attacks against other AWS services that can trigger your function. These weaknesses give threat actors the opportunity to execute code while making it difficult for you to detect them. Practicing robust code reviews, static and dynamic source security scans (static/dynamic application security testing [SAST/DAST]), and fuzz testing can help you uncover potential weaknesses.

Improper Access Control

Missing or incorrectly configured access controls allow unauthorized users to invoke your functions. You can prevent this by creating appropriate AWS Identity and Access Management (IAM) policies to precisely define which users and applications can interact with your functions.

It's also important to assign your functions a narrowly scoped execution role. This configures the AWS resources that your function's code can access using the credentials that are injected into its environment. Functions should be assigned the minimum privileges they require, such as permission to access files in a specific Amazon Simple Storage Service (Amazon S3) bucket but not any other type of resource.

Secure AWS Lambda: Protect Your Functions with a Mesh Network

Lambda security issues often stem from functions being unnecessarily publicly exposed. Although this can be unavoidable in some situations, many functions interact only with other components in your stack. Setting up private networking flows for these functions removes the need to expose them on the internet, reducing your attack surface.

The best way to securely connect your functions to your other resources is with an encrypted mesh network like NetBird. NetBird lets you link your infrastructure together using a zero-config private WireGuard network that works across cloud, serverless, and on-premise infrastructure.

Joining serverless environments like AWS Lambda to a mesh network has traditionally been difficult because you can't directly access the network interfaces on the hosts that run your functions. The NetBird netstack mode addresses this by providing a simulated TUN device and a SOCKS5 proxy that targets that device. This allows your Lambda function to access other services in your NetBird network via the proxy.

How to Securely Run Lambda Functions with NetBird

To use NetBird with your Lambda functions, you need to follow a few simple steps:

Write your function's code: Create your function in the usual way, including any network connections you require. However, you should ensure the networking libraries you use are configured to connect using the SOCKS5 proxy that NetBird provides.
Create a Docker image for your function: Your Lambda functions must be deployed as container images so the NetBird binary can be run alongside your code. Within your container image's build, you should select a relevant AWS Lambda image as your base image, then add your code and the /go/bin/netbird binary from the NetBird official image.
Enable the NetBird netstack mode: You need to set the NB_USE_NETSTACK_MODE and NB_FOREGROUND_MODE environment variables inside your container image to ensure NetBird is correctly configured for in-container netstack operation.
Create a command script that starts NetBird before your function runs: You can achieve this by setting a custom script as your container's entry point. It should start the NetBird binary and then invoke your code using the AWS Lambda platform binary.
Set your NetBird setup key as a Lambda environment variable: Before running your function, you should set the NB_SETUP_KEY environment variable using the Lambda console or API. Setup keys connect your apps to your NetBird network. You can generate one by logging into your NetBird account.

After following these steps, you can deploy your Lambda project. NetBird will register your function in the network and assign it a private NetBird IP address.
You can use this IP to call your function. It executes inside the container using the image you've created.
Because the NetBird binary is configured to run inside the container, too, your code can access the other resources in
your network without being exposed to the public internet. You can even connect your Lambda to on-premise resources like databases or legacy systems,
or even multiple serverless functions running in different cloud providers.

Check out the NetBird documentation for more information on how to set up your Lambda functions with NetBird.

Best Practices for AWS Lambda Security

AWS Lambda simplifies cloud deployment by letting you launch services as serverless functions that run on demand with high availability and scalability. Although this can help you deliver code more quickly and cost-effectively, it also makes you vulnerable to new security threats if you don't properly harden your AWS accounts and the functions within them. Here's a recap of the best practices you should follow to defend your serverless functions against these risks:

Configure correct IAM policies to restrict access to your functions.
Allocate appropriate resources to your Lambda functions to avoid resource exhaustion or overprovisioning.
Practice secure coding methods, including proper input sanitization and safe supply chain hygiene.
Ensure that activities in functions are observable through logs and error reports.
Avoid exposing functions on the public internet.
Connect functions to other infrastructure using a private VPN.

These steps ensure your Lambda functions are performant, safe, and secure, letting you fully benefit from the development efficiency improvements that serverless computing allows.

Using NetBird for Kubernetes Access

Misha Bragin — Mon, 29 Apr 2024 18:24:12 +0000

Securing access to your Kubernetes clusters is crucial as inadequate security measures can lead to unauthorized access and potential data breaches. However, navigating the complexities of Kubernetes access security, especially when setting up strong authentication, authorization, and network policies, can be challenging.

NetBird simplifies Kubernetes access with its zero-configuration approach, leveraging WireGuard's simplicity and strength. It seamlessly integrates with various tools, offering transparency and high reliability as an open source solution.

In this article, you'll learn how to set up the NetBird CLI to ensure a secure connection to a Kubernetes cluster on the Google Cloud Platform (GCP), complete with a fail-safe route for uninterrupted access.

Using NetBird for Kubernetes Access

In this tutorial, you'll follow these steps to enable secure access to your Kubernetes cluster using NetBird:

Set up a NetBird agent on your local machine.
Create NetBird setup keys to be used in the Kubernetes cluster.
Set up a remote Kubernetes cluster. For this tutorial, Google Kubernetes Engine (GKE) was chosen; however, feel free to use any remote Kubernetes cluster.
Deploy the setup key in the remote Kubernetes cluster.
Configure a highly available (HA) network route in NetBird for uninterrupted access to the Kubernetes cluster.
Test secure access from the local client to the remote Kubernetes cluster.

Please note that to follow this tutorial, you need kubectl and gcloud command line tools installed and properly configured on your local machine.

With the procedure outlined, let's get started.

Set Up the NetBird Agent

If you haven't already, sign up for a free NetBird account. After confirming your email,
you'll receive a message with instructions on how to install the NetBird agent on your local machine.
This agent includes NetBird's CLI, which is used to control it.

Follow the instructions for your operating system and finish by authorizing the NetBird app to access the NetBird dashboard using the command sudo netbird up:

You should see your local machine in the Peers dashboard, which means that it's now connected to the NetBird peer-to-peer (P2P) network:

Create Setup Keys

Now that you're connected to the network, it's time to add a Kubernetes cluster to this network. To do that, you need to create a setup key that your Kubernetes cluster can use to authenticate with NetBird.

Click on the Setup Keys tab on the left-hand side of your screen and then click on the Create Setup Key button:

A pop-up window opens where you can name the key, set the maximum number of peers that can use it, and determine its expiration date.
Additionally, you can also select auto-assigned groups.
With this option, any peer that registers with the key will be automatically added to these groups, meaning that the access control
rules set for these groups will automatically apply to them. Set it to gke-cluster, it will help you to avoid adding
each peer to the group manually later in the tutorial.

You'll also see two toggle switches there: one for making the key reusable and another to mark a peer as
ephemeral.
Reusable setup keys are useful when there is no human behind a machine that can use SSO and MFA, e.g.,
to associate servers with a NetBird account,
so they can authenticate to the network automatically. On the other hand, ephemeral keys automatically remove machines
from NetBird if they are offline for more than 10 minutes. This feature helps prevent offline machines from cluttering
the NetBird peer dashboard, such as when a Kubernetes pod gets replaced by a new one.

The following screenshot shows the creation of an ephemeral key:

Likewise, this screenshot shows how to create a reusable key:

Given the reasons mentioned earlier, using ephemeral keys for this tutorial is recommended. This is because Kubernetes
might restart services, causing machines to go offline.

Set Up a Remote Kubernetes Cluster

To ensure a highly available network route with NetBird, there are some considerations to keep in mind when deploying the GKE cluster:

You should create, at a bare minimum, a regional cluster or a multizonal cluster with nodes distributed across multiple zones within a region to ensure that the cluster is resilient to zonal failures.
You need to deploy at least three nodes to ensure high availability.
You are highly recommended to enable cluster autoscaling to ensure that there are always at least three nodes running.

You can use the Google Cloud console UI to set up a Kubernetes cluster like the one described earlier, or you can adapt this gcloud command to fit your needs:



gcloud container --project "YOUR_PROJECT_ID" clusters create "netbird-cluster-1" \
--no-enable-basic-auth --release-channel "regular" \
--machine-type "e2-small" --image-type "COS_CONTAINERD" --disk-type "pd-standard" \
--disk-size "100" --metadata disable-legacy-endpoints=true \
--num-nodes "1" --logging=SYSTEM,WORKLOAD --monitoring=SYSTEM --enable-ip-alias \
--enable-autoscaling --min-nodes "1" --max-nodes "2" \
--network "projects/YOUR_PROJECT_ID/global/networks/default" \
--subnetwork "projects/YOUR_PROJECT_ID/regions/us-east1/subnetworks/default" \
--no-enable-intra-node-visibility --default-max-pods-per-node "110" \
--security-posture=standard --workload-vulnerability-scanning=disabled \
--no-enable-master-authorized-networks --addons HorizontalPodAutoscaling,HttpLoadBalancing,GcePersistentDiskCsiDriver \
--enable-autoupgrade --enable-autorepair --max-surge-upgrade 1 --max-unavailable-upgrade 0 \
--enable-managed-prometheus --enable-shielded-nodes \
--region "us-east1" --node-locations "us-east1-b,us-east1-c,us-east1-d"

Replace YOUR_PROJECT_ID with the ID of your project. Additionally, replace the region us-east1 and the zones us-east1-b, us-east1-c, and us-east1-d according to your preferences.

If you want to create a cluster with a specific version, use the --cluster-version flag to match the latest available version in your region.

Pay attention to the instance type configuration. In this case, e2-small guarantees NetBird has sufficient CPU and memory resources. --num-nodes is set to 1 since it deploys one node per zone (three nodes in total), and the --enable-autoscaling flag is set to allow a maximum of two nodes per zone or a maximum of six nodes in total.

Once the cluster is deployed, log in to the Google Cloud console and check that everything is working as expected (cluster status is Ok and status of the nodes' is Ready):

You can also verify the nodes using kubectl from your local machine:



$ kubectl get nodes
NAME                                               STATUS   ROLES    AGE     VERSION
gke-netbird-cluster-1-default-pool-6969d6cf-w5fd   Ready    <none>   5m13s   v1.27.8-gke.1067004
gke-netbird-cluster-1-default-pool-7ddbb961-8rkf   Ready    <none>   5m14s   v1.27.8-gke.1067004
gke-netbird-cluster-1-default-pool-a7de6516-rbb0   Ready    <none>   5m14s   v1.27.8-gke.1067004

With the GKE cluster up and running, it's time to add the NetBird ephemeral key.

Add the NetBird Ephemeral Setup Key to the Kubernetes Cluster

To register your Kubernetes cluster to the NetBird secure P2P network, create a YAML file on your local machine called netbird-client.yaml and paste the following content in it:



apiVersion: apps/v1
kind: Deployment
metadata:
  name: netbird
spec:
  selector:
    matchLabels:
      app: netbird
  replicas: 3
  template:
    metadata:
      labels:
        app: netbird
    spec:
      containers:
        - name: netbird
          image: netbirdio/netbird:latest
          env:
            - name: NB_SETUP_KEY
              value: <YOUR_SETUP_KEY>
            - name: NB_HOSTNAME
              value: "netbird-cluster-1"
            - name: NB_LOG_LEVEL
              value: "info"
          volumeMounts:
            - name: netbird-client
              mountPath: /etc/netbird
          resources:
            requests:
              memory: "128Mi"
              cpu: "500m"
          securityContext:
            privileged: true
            runAsUser: 0
            runAsGroup: 0
            capabilities:
              add:
                - NET_ADMIN
                - NET_RESOURCE
                - SYS_ADMIN
      volumes:
        - name: netbird-client
          emptyDir: {}

Let's break down the key arguments of this deployment:

replicas: 3 ensures that at least one replica is deployed per node and is necessary to secure HA.
image: netbirdio/netbird:latest uses the latest available image. For production, it's best practice to use a specific version to avoid conflicts during updates.
The requests parameter ensures that NetBird has a minimum guaranteed amount of resources to run properly. It's best practice to assign requests and/or limits to any app or service running on your cluster, which includes the NetBird client.
privileged: true is required since, generally, modifying the routing table requires root permissions.

Remember to replace YOUR_SETUP_KEY with the previously created ephemeral setup key. Once you're ready, add the ephemeral key to the cluster by running the following command:



kubectl apply -f netbird-client.yaml

You can check the status of the deployment using kubectl get pods:



kubectl get pods
NAME                       READY   STATUS    RESTARTS   AGE
netbird-7dc884d77f-285dh   1/1     Running   0          96s
netbird-7dc884d77f-cfq69   1/1     Running   0          96s
netbird-7dc884d77f-x56nh   1/1     Running   0          96s

At this point, if you go to the NetBird Peers tab, you should see the GKE nodes:

With your local machine and GKE nodes now part of the NetBird P2P network, you're ready to establish a secure HA network route for communication.

Set Up an HA Network Route in NetBird

Simply having all GKE nodes on the same peer-to-peer network doesn't guarantee accessibility. For instance, if you attempt to ping a node, you'll receive a "request timeout…" message. This happens because you need to set up a network route and define an access policy before you can connect to the nodes.

To facilitate the process of creating network routes and access policies, NetBird lets you create peer groups.
Do you remember that you created and assigned the gke-cluster group to the GKE nodes when you created the ephemeral key?
If you haven't done this then, you can create and assign the group manually to the peers now.

To do so, click on the first GKE node in the Peers view. This takes you to a screen with the peer details. On the right side, you should see Assigned Groups:

Start typing a name (ie gke-cluster) to create a new group. After you've entered the name, click Save Changes to apply the new configuration:

Repeat the procedure for the remaining two GKE nodes; remember to choose the same group and save the changes.

Once finished, hover the mouse pointer over the GROUPS column on any of the nodes to verify that they belong to the desired group:

Similarly, to simplify setting a route, you can assign a group to your local machine, such as local:

With the peer groups ready, it's time to move to the Network Routes tab, which you can find on the left-hand side of your screen:

Before setting up your first network route, you need to determine which network you wish to connect with. For instance, if you want to access pods and services in your Kubernetes cluster, you'll need the GKE cluster Pod IPv4 and IPv4 service ranges. To find these, navigate to the Networking section in the Google Cloud console:

Click on the Add Route button, and a pop-up opens, allowing you to create a new route.

Input the Network Range, then click on Peer Group and choose gke-cluster (or the group you've established for your nodes). Under Distribution Groups, pick the local group you just created to make this route known to your local machine:

In the Name & Description tab, provide a name (network identifier) for the route and click Add Route to finalize its creation:

Following is an example of two networks: one for the GKE pods and another for the GKE services. Note that the HIGH AVAILABILITY column shows 3 Peer(s) in green to indicate that it is a highly available route:

While these routes create a link between your local machine and the GKE cluster, you still need one more component to enable communication: an access control policy (ACP).

Set Up an Access Control Policy to Access Your Kubernetes Cluster

To create an ACP, go to Access Control > Policies and click on Add Policy. A pop-up appears where you can define a new ACP. Select the local group as Source and the gke-cluster group as Destination. This allows bidirectional communication between both peers:

Configuring security posture checks is beyond the scope of this guide, but in a nutshell, the Posture Checks tab lets you enhance access control by blocking peers that fail to meet certain conditions, such as client version, country of origin, and operating system.
If you are interested in this and other Zero Trust Security features, check the Open-Source Zero Trust Networking Guide.

To create the new policy, go to the Name & Description tab and enter an appropriate name for the rule (eg Local to GKE):

To finish setting up the new ACP, select Add Policy.

The following are two example policies: Local to GKE and the NetBird Default policy. In this example, the default policy has been turned off, so there's only Local to GKE active:

Congratulations! You just configured secure access to Kubernetes using NetBird P2P. To check the connection, navigate to the Peers tab and take note of the IP address or domain assigned by NetBird to any of the nodes.

The following image shows the response when pinging one of the GKE nodes:



$ ping -c 3 netbird-cluster-1.netbird.cloud

PING netbird-cluster-1.netbird.cloud (100.74.176.22): 56 data bytes

64 bytes from 100.74.176.22: icmp_seq=0 ttl=64 time=106.287 ms

64 bytes from 100.74.176.22: icmp_seq=1 ttl=64 time=107.516 ms

64 bytes from 100.74.176.22: icmp_seq=2 ttl=64 time=105.674 ms

--- netbird-cluster-1.netbird.cloud ping statistics ---

3 packets transmitted, 3 packets received, 0.0% packet loss

round-trip min/avg/max/stddev = 105.674/106.492/107.516/0.766 ms

Conclusion

In this tutorial, you installed the NetBird CLI, configured a Kubernetes cluster on GCP, integrated a NetBird ephemeral key, created a high-availability route, and set an access control policy for secure local access.

Take your network management to the next level—explore more features and enhance your setup with the NetBird cutting-edge peer-to-peer network.

Serverless Security Vulnerabilities and Best Practices to Mitigate Them

Misha Bragin — Tue, 23 Apr 2024 15:49:19 +0000

The cloud has revolutionized application development, and serverless computing is its latest evolution. It offers an entire paradigm shift, allowing developers to focus solely on code functionality without having to manage servers. However, this convenience comes with a twist: new security challenges.

Traditional security practices are centered on securing underlying infrastructure, but serverless computing lacks dedicated servers and introduces a new attack surface. Malicious actors can exploit vulnerabilities in code, permissions, and event data to compromise serverless applications. This can lead to data breaches, unauthorized access, and financial loss.

In this article, you'll learn all about security vulnerabilities that can occur in serverless environments and how to prevent them. By the end of the article, you'll be able to build and deploy secure serverless applications confidently.

Why You Need to Secure Your Serverless Functions

Serverless security goes beyond traditional application security. It focuses on protecting the code within your serverless functions, the data those functions handle, and the overall execution environment provided by the cloud platform. This shift in focus is crucial because serverless functions often become the primary targets for attackers due to accessibility, heavy reliance on third-party code/services, shared resources, and limited control over infrastructure.

If your serverless functions process any personally identifiable information (PII), financial transactions, or health information or manage intellectual property, even a minor security breach can have devastating consequences. Strong serverless security safeguards this sensitive information and ensures the integrity of your functions. Additionally, many applications must comply with regulations, such as the General Data Protection Regulation (GDPR) or the California Consumer Privacy Act (CCPA). That means securing your functions and your data is a necessity.

In addition, serverless applications often interact with data residing outside the cloud platform, such as on-premise databases or storage systems. This hybrid environment creates additional security considerations. For example, a function might need to securely access a customer database located in your corporate data center. Mitigating unauthorized access to this data requires robust security measures at the function level to ensure proper authentication and authorization protocols are followed during communication.

Data migration projects are another compelling reason to prioritize serverless security. Imagine a scenario where you're migrating a workload (functions) to the cloud while the data (database) remains on-premise. In this case, ensuring secure communication between the functions and the on-premise data is critical. Serverless security best practices, such as least privilege access and data encryption, can help prevent unauthorized access during the migration process.

Best Practices to Secure Your Serverless Functions

Now that you know why you need to secure your serverless functions, let's take a look at a few best practices that you can implement to help you secure them.

Implement an Authentication and Authorization Solution That Integrates with Your Function

Authentication (authn) and authorization (authz) are the cornerstones of web application security. Authentication verifies a user's identity, while authorization determines their access rights to specific functions or resources. In a serverless environment, these mechanisms are typically integrated via an API gateway, which acts as the entry point for your functions, or through identity and access management (IAM) rules specific to a cloud platform that allow the functions to interact with other cloud resources.

Without proper controls, any person or service could trigger your functions and steal data or disrupt operations. Authentication ensures that only authorized users can interact with your functions, while authorization restricts their actions based on predefined roles or permissions.

Depending on the cloud platform, there are multiple ways you could set up authn/authz for your serverless functions. For instance, Amazon Web Services (AWS) provides user authentication through Amazon Cognito. The API gateway can be configured to integrate with Cognito to require users to sign in before invoking any Lambda functions. Cognito verifies user identities and passes relevant tokens to your functions for authorization decisions.

Additionally, AWS IAM allows you to define roles with specific permissions. You can associate an IAM role with your Lambda function, granting it access to specific resources, such as Amazon Simple Storage Service (Amazon S3) buckets or DynamoDB tables.

Perform Input Validation and Sanitization

Serverless functions often accept input from various sources, including user requests, API calls, and event triggers. Malicious actors can exploit improperly validated or sanitized input to inject code or manipulate data. This is why input validation and sanitization are critical security practices.

Unvalidated or unsanitized input can lead to vulnerabilities, such as SQL injection, cross-site scripting (XSS), and command injection. These attacks can compromise your functions, steal data, and disrupt operations.

Following are a few ways you can implement proper input sanitization and validation in your serverless functions:

Validate input types and formats: Ensure user input conforms to expected data types (eg integer or string) and formats (eg email address or date). AWS Lambda supports libraries such as Ajv or native JavaScript functions for data type validation.
Sanitize user input: Remove potentially harmful characters or code from user input before processing it. You can consider setting up a security handler using Vanadium to add custom, reusable logic for handling input validation and sanitization.
Encode special characters: Make sure to encode special characters when sending user input to prevent them from being misinterpreted. For instance, if you're sending input to functions through HTTP parameters, then you need to encode & and = characters because they have special meanings in URLs. JavaScript's encodeURIComponent and encodeURI functions can be used for this purpose.

Validate Third-Party Library Dependencies in Your Code

When writing serverless apps, developers often use third-party libraries to implement commonly used solutions and logic. However, the convenience of third-party libraries can come with hidden security risks. If not carefully vetted and maintained, these libraries can introduce vulnerabilities and create openings for attackers. A compromised library can be used to steal data, inject malicious code, or disrupt your functions. Validating your third-party dependencies is essential for robust serverless security.

Following are a few tips you can implement to help you carefully validate and pick the right library dependencies for your serverless app:

Use trusted sources and stay updated: Download libraries only from official repositories, such as npm, or maintain them in private repositories with proper access controls. Additionally, keep your libraries up-to-date to benefit from security patches.
Review library security reports: Many popular libraries have publicly available security reports detailing known vulnerabilities. Review these reports before using the library and consider alternatives if critical vulnerabilities exist.
Leverage dependency scanning tools: Several tools, such as Snyk or npm-audit, can scan your Lambda function dependencies for known vulnerabilities. These tools can identify potential security risks and recommend mitigation steps.

While libraries offer convenience, make sure you use only essential libraries to reduce your attack surface and simplify security management. Additionally, consider building custom functionality if a readily available library has a questionable security track record and there are no secure alternatives.

Enable Centralized Monitoring and Logging

Centralized monitoring and logging provide a comprehensive view of your functions' health and security posture and can help you maintain a watchful eye over your serverless functions. Without proper monitoring, you could miss critical security events, such as unauthorized access attempts or function errors. Centralized logs allow you to analyze function behavior, identify anomalies, and detect potential threats.

Most cloud platform vendors offer services to help you with active monitoring and alerts. For instance, Amazon CloudWatch provides a centralized logging service for your AWS Lambda functions. You can configure your Lambda functions to send logs to CloudWatch, allowing you to aggregate and analyze logs from all your functions in one place.

Several third-party tools, such as Splunk or Sumo Logic, can be integrated with CloudWatch to provide advanced log analytics and visualization. These tools can help you identify trends, filter logs for specific events, and set up alerts for suspicious activity.

CloudWatch also provides monitoring capabilities. Lambda automatically reports metrics related to basic performance, duration, and resource utilization through CloudWatch, and you can configure Lambda functions to emit custom metrics that provide insights into statistics local to the app domain. You can then use these metrics to identify potential performance bottlenecks and diagnose function errors.

Review Your Security Configurations to Remove Overly Permissive or Stale Configurations

A large part of web security hinges on proper configuration management. Permissions granted to functions and access controls for resources need to be continuously reviewed and adjusted to minimize the attack surface.

Overly permissive configurations or outdated permissions can create security gaps. An unused function with excessive IAM permissions or an API gateway endpoint with unrestricted access can be exploited.

Here's how you can ensure your serverless functions benefit from secure and up-to-date configurations:

Implement least privilege: Grant your serverless functions only the minimum permissions required to perform their tasks. Avoid using wildcards or overly broad IAM policies. The principle of least privilege minimizes the potential damage if a function is compromised.
Review and remove unused resources: Periodically review IAM roles, API gateway permissions, and other security configurations associated with your serverless functions. Identify and remove unused resources to eliminate potential attack vectors and streamline your security posture.
Utilize infrastructure as code (IaC): Leverage IaC tools, such as Terraform or AWS CloudFormation (in the case of AWS), to manage your serverless infrastructure configurations. IaC allows you to version control your configurations, track changes, and automate deployments. This ensures consistency and reduces the risk of manual configuration errors.
Enable security best practices: Many cloud providers offer security best practices recommendations for serverless functions. In AWS, the AWS Security Hub can identify potential security issues in your Lambda function configurations and suggest remediation steps.

Consult the OWASP Serverless Top Ten Documentation

The Open Worldwide Application Security Project (OWASP) is a renowned community for web application security. The OWASP Serverless Top Ten project addresses specific security challenges for serverless environments. This resource catalogs the ten most critical security risks specific to serverless functions. Understanding these threats allows you to prioritize your security efforts and implement effective mitigation strategies.

The documentation discusses the various ways in which injection attacks can impact a serverless app and lists ways that you can safeguard your functions against them. Similarly, it talks about broken authentication, sensitive data exposure, XML external entities, broken access control, security misconfiguration, XSS, insecure deserialization, use of components with known vulnerabilities, and insufficient logging and monitoring in depth.

Conclusion

Serverless security demands a proactive approach. You need to understand the unique threat landscape and implement best practices, such as robust authentication/authorization, meticulous input validation, and secure coding practices, to help you reduce your attack surface. Remember to leverage cloud provider security features, monitor your functions vigilantly, and keep configurations up-to-date.

For a deeper dive into serverless vulnerabilities, explore the OWASP Serverless Top Ten documentation. When you prioritize serverless security, you can build and deploy applications with confidence, ensuring the integrity of your data and the smooth operation of your serverless environment.

NetBird secures access and connectivity to serverless functions on platforms like AWS and Microsoft Azure. NetBird leverages netstack from the gVisor Go package (part of wireguard-go) and enables WireGuard to run entirely in the user space. This method alleviates the need for network-level or kernel-level access.

To learn how to configure NetBird, check out this Python database access example on Azure functions.

Using eBPF and XDP to Share Default DNS Port Between Multiple Resolvers

Misha Bragin — Tue, 23 Apr 2024 15:41:32 +0000

Have you tried configuring a local DNS resolver to use a port different from the default one?
Changing port 53 can be tricky since the DNS protocol is usually bound to this port.
This is especially true if you need the software you develop to support as many operating systems as possible with various versions,
ensuring it works on different platforms, too.

The NetBird team recently faced this and other challenges while working on the DNS feature that makes managing DNS in private networks easy.
We spent a few days on a solution involving Go, eBPF, and XDP "magic" that allows sharing a single port between multiple DNS resolver processes.
We also created a DNS manager for WireGuard® networks that universally works on macOS, Linux, Windows, Docker, and mobile phones.
Now is the time to share our journey. Traditionally, as with the whole NetBird platform, the code is open-source.

About NetBird

Before jumping into the details, it is worth mentioning why we've dived into this topic and how NetBird simplifies DNS management in private networks.

NetBird is a zero-configuration overlay network and remote access solution that automatically connects your servers, containers,
cloud resources, and remote teams over an encrypted point-to-point WireGuard tunnel.
There is also something special about NetBird that gave us a "little" headache while working on the DNS support and forced us to use XDP.
I'm talking about kernel WireGuard. If available, NetBird uses the kernel WireGuard module shipped with Linux.
Otherwise, it falls back to the userspace wireguard-go implementation,
a common case for non-Linux operating systems.

To minimize the configuration effort on the administrator side, NetBird allocates network IPs from the CGNAT 100.64.0.0/10
range and distributes them to machines from a central place. Then, it configures a WireGuard interface locally on every machine,
assigning IPs to create direct connections.

Here is how the WireGuard interface looks on my Ubuntu laptop after NetBird has done its configuration job:

misha@misha-linux:~$ ip address show wt0
5: wt0: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1280 qdisc noqueue state UNKNOWN group default qlen 1000
    link/none
    inet 100.109.71.139/16 brd 100.109.255.255 scope global wt0
       valid_lft forever preferred_lft forever

NetBird assigned the private IP 100.127.136.241 to the WireGuard interface wt0 on my machine. Similarly, it assigned private IPs to remote machines that my laptop is allowed to connect to (allowed ips):

misha@misha-linux:~$ sudo wg show
interface: wt0
public key: LAo1oYyliEnvENFRZJSVb93K1kOZFu627/KSxSYjWAw=
private key: (hidden)
listening port: 51820

peer: HzjQ27XQHhm3Ky2jrCDOOF63hgop/GS6zfzYEQmH6VA=
endpoint: 35.191.145.11:62519
allowed ips: 100.127.78.76/32
latest handshake: 24 seconds ago
transfer: 124 B received, 180 B sent
persistent keepalive: every 25 seconds

peer: Fz30t36HImRdt1nwVslVFxy2Q96OEB2SwCFYGeIHdmA=
endpoint: 34.173.121.152:35895
allowed ips: 100.127.58.115/32, 172.17.0.0/16
latest handshake: 25 seconds ago
transfer: 124 B received, 180 B sent
persistent keepalive: every 25 seconds

Need for DNS in private networks

Now, what about DNS? Wouldn't it be much more convenient to have a meaningful and easily memorable name instead of the
100.127.16.22 IP address to access a privately hosted Postgres database hidden behind this IP?
To address this, NetBird automatically assigns a domain name to each peer in the private netbird.cloud space that one can use for remote access.

Running the netbird status -d command shows that my machine has connected to postgres.netbird.cloud and berlin-office.netbird.cloud:

misha@misha-linux:~$ netbird status -d
Peers detail:
postgres.netbird.cloud:
NetBird IP: 100.127.16.22
Public key: 99W1Y7p3QPesQpEYu7i0mXlvo+jBhv4DzvY7apPDDBM=
Status: Connected
-- detail --
Connection type:P2P
Last connection update: 2023-12-13 11:47:56

berlin-office.netbird.cloud:
NetBird IP: 100.127.58.115
Public key: Fz30t36HImRdt1nwVslVFxy2Q96OEB2SwCFYGeIHdmA=
Status: Connected
-- detail --
Connection type: P2P
Last connection update: 2023-12-13 11:43:32

The ping command successfully resolves postgres.netbird.cloud to 100.127.16.22:

misha@misha-linux:~$ ping -c 3 postgres.netbird.cloud
PING postgres.netbird.cloud (100.127.16.22) 56(84) bytes of data.
64 bytes from 100.127.16.22: icmp_seq=1 ttl=64 time=34.6 ms
64 bytes from 100.127.16.22: icmp_seq=2 ttl=64 time=35.6 ms
64 bytes from 100.127.16.22: icmp_seq=3 ttl=64 time=33.0 ms

Local DNS resolution implementation

How does NetBird do it so that DNS resolution works on Linux, Mac, Windows, mobile phones, and even inside Docker? While
working on the feature, we had a few options. One was to set up a centralized DNS server, e.g., dns.netbird.io,
and configure local resolvers on all connected machines to point to it.
However, such an approach brings scalability, performance, and, more importantly, privacy issues.
Like network traffic, we prefer our users' DNS queries not to go through our servers.

Another approach is to modify the local /etc/hosts file on every machine, specifying the remote machines' name-IP pairs.
The hosts file can quickly outgrow when dealing with large networks, and who would like it if NetBird modified this file?

We took the locality approach and avoided significant system configuration changes. Every NetBird agent starts an embedded
DNS resolver that holds the NetBird name-IP pairs of accessible machines in memory and resolves the names when requested.
The centralized NetBird management service sends DNS updates (essentially name-IP pairs) through the control channel
when new machines join or leave the network or the administrator changes DNS settings.

We applied this approach to all supported operating systems. However, there is a caveat.
The NetBird agent must also configure the machine's operating system to point all netbird.cloud queries to the local NetBird resolver.

Configuring DNS management varies from operating system to operating system and even version to version.
Add kernel WireGuard to it, and you will get some interesting implementation differences. Read on, and you will finally get to the port issue!

Configuring DNS on Linux

Linux is the most used OS in NetBird. The great variety of Linux flavors and versions make the administration,
for lack of a better word, unpleasant. DNS configuration is not the exception - Linux offers different ways to do it.
Depending on the system setup, you could use the NetworkManager service,
systemd-resolved service,resolvconf command line tool, and a direct modification of the resolv.conf file.

Which one does NetBird use? As mentioned earlier, we avoid causing significant changes to the system configuration.
Therefore, NetBird checks the /etc/resolv.conf file that usually contains a comment indicating how the system manages DNS:

misha@misha-linux:~$ head -n 2 /etc/resolv.conf
# This is /run/systemd/resolve/stub-resolv.conf managed by man:systemd-resolved(8).
# Do not edit.

My Ubuntu system uses the systemd-resolved service. There is also a clear warning not to edit; we follow the advice.

With a bunch of "if a string contains" conditions, NetBird chooses the right manager.

After picking the manager, NetBird starts configuring DNS.
It uses D-Bus for communicating with NetworkManager and systemd-resolved, os.exec with resolvconf, and os.WriteFile for a direct /etc/resolv.conf file modification:

If you are familiar with the Go programming language, look at the open-sourced code calling the systemd-resolved service via D-Bus.

Below is the command-line equivalent of the Go code that uses Ubuntu's default bustctl utility to call D-Bus:

busctl call org.freedesktop.resolve1 \
  /org/freedesktop/resolve1/link/_312 \
  org.freedesktop.resolve1.Link \
  SetDNS 'a(iay)' 1 2 \
  4 100 127 136 241

What does the configuration result look like? When I run the resolvectl status command on my machine,
the applied DNS configuration appears as follows:

# wt0 is the WireGuard interface that NetBird created on the machine
misha@misha-linux:~$ resolvectl status wt0
Link 12 (wt0)
    Current Scopes: DNS
         Protocols: +DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
Current DNS Server: 100.127.136.241
       DNS Servers: 100.127.136.241
        DNS Domain: netbird.cloud

The agent reused its NetBird private IP 100.127.136.241 and configured the system's DNS to resolve netbird.cloud
with the embedded resolver listening on the default DNS port 53.

Port 53 issue

The setup described above looks straightforward, doesn't it? The NetBird agent started a local resolver,
received a configuration from the management server, and applied it to the system. However, there are always exceptions.
Some of them are very common. Other processes like dnsmasq, Unbound, or Pi-hole might already occupy the default DNS port.
On top of that, some systems won't allow administrators to specify a custom port. Support for custom DNS ports was added
to the systemd-resolved service only in 2020,
while resolv.conf file doesn't support it to date. What to do in this situation?
NetBird pursues the goal of a networking platform that works anywhere, universally, and without complex configurations.
Therefore, we had to apply some logic to work around the issue and came up with two solutions to most of the corner cases.

Solution: unassigned loopback address

The NetBird-embedded DNS resolver tries to listen for incoming DNS queries on the private NetBird IP or on the loopback address and port 53,
e.g., 100.127.136.241:53 and 127.0.0.1:53, respectively. As we discovered, this may not be possible due to some other process occupying port 53.
NetBird yields to an unassigned loopback address, 127.0.0.153:53 in this case.
The Linux operating system will allow a combination of an unassigned loopback address and port even though it is occupied.

Below is a small Go program that simulates the situation and tries listening on 127.0.0.1:53 and then on 127.0.0.153:53:

func main() {
    // array of IP:port combinations
    addresses := []string{"127.0.0.1:53", "127.0.0.153:53"}

    // create a listener for each address
    for _, address := range addresses {
       time.Sleep(time.Second)
       go func(addr string) {
          packetConn, err := net.ListenPacket("udp", addr)
          if err != nil {
             fmt.Printf("failed creating UDP listener on %s: %v\n", addr, err)
             os.Exit(1)
          }
          defer packetConn.Close()
          fmt.Printf("successfully created UDP listener on %s\n", addr)

          // handle packets
          for {
             buffer := make([]byte, 1024)
             _, _, err = packetConn.ReadFrom(buffer)
             if err != nil {
                fmt.Printf("error reading packet: %v\n", err)
                continue
             }
          }
       }(address)
    }

    // Keep the main goroutine running
    select {}
}

After compiling and running the code, we see that it worked without issues:

misha@misha-linux: go build -o test && sudo ./test
successfully created a UDP listener on 127.0.0.1:53
successfully created a UDP listener on 127.0.0.153:53

Netstat also indicates that there are two go processes listening on port 53:

misha@misha-linux:~$ sudo netstat -tulpn | grep 53
udp        0      0 127.0.0.153:53          0.0.0.0:*          24356/./test
udp        0      0 127.0.0.1:53            0.0.0.0:*          24356/./test

By the way, the systemd-resolved service has a similar behavior and listens on 127.0.0.53:53.

That was a theory, but what does it look like in real life?

100.127.136.241:53 address, a private NetBird IP, may not be available because of the dnsmasq service that listens
on all interfaces by default (omitting ipv6 for simplicity):

tcp        0      0 172.17.0.1:53           0.0.0.0:*         LISTEN      26136/dnsmasq
tcp        0      0 127.0.0.1:53            0.0.0.0:*         LISTEN      26136/dnsmasq
tcp        0      0 100.127.136.241:53      0.0.0.0:*         LISTEN      26136/dnsmasq
udp        0      0 127.0.0.1:53            0.0.0.0:*                     26136/dnsmasq
udp        0      0 100.127.136.241:53      0.0.0.0:*                     26136/dnsmasq
udp        0      0 172.17.0.1:53           0.0.0.0:*                     26136/dnsmasq

How about 127.0.0.1:53? Similar to dnsmasq another DNS service unbound occupies this address:

tcp        0      0 127.0.0.1:8953          0.0.0.0:*         LISTEN      27023/unbound
tcp        0      0 127.0.0.1:53            0.0.0.0:*         LISTEN      27023/unbound
udp        0      0 127.0.0.1:53            0.0.0.0:*                     27023/unbound

Switching to 127.0.0.153:53 should do the trick in most cases. But what if some process listens on all interfaces?

When a process listens on all interfaces, in other words, 0.0.0.0:53,
Linux won't allow using unassigned loopback addresses.
We can quickly test this scenario by modifying the address parameter in the Go testing code by placing 0.0.0.0:53 first:

addresses := []string{"0.0.0.0:53", "127.0.0.1:53", "127.0.0.153:53"}

The code fails with the address already in use error:

misha@misha-linux: go build -o test && sudo ./test
successfully created a UDP listener on 0.0.0.0:53
failed creating a UDP listener on 127.0.0.1:53: bind: address already in use

That could happen when network administrators apply custom configurations to the system.
NetBird uses eBPF with XDP (eXpress Data Path) to work around this issue and share the port with the other process.

Solution: port forwarding with XDP and eBPF

From the documentation: XDP is a framework that enables high-speed packet processing within eBPF applications.
To enable faster response to network operations,
XDP runs a eBPF program as soon as possible, immediately as the network interface receives a packet. Here is how we use it in NetBird:

Shortly, the agent initiates an embedded NetBird DNS listener on a custom port 5053 and NetBird IP 100.127.136.241,
that the NetBird management service assigned to the peer.
However, due to issues encountered when using custom ports, the agent configures the system to reroute all
netbird.cloud queries to a "fake" resolver address using the peer's NetBird IP 100.127.136.241:53.
This address is referred to as "fake" because there is no active listener on it. Subsequently,
the eBPF program intercepts network packets destined for the fake address and
forwards them to port 5053, where the real embedded NetBird DNS resolver listens.

We developed the eBPF program in C with a core function xdp_dns_fwd shown below and attached it
to the loopback interface lo with XDP.
This function modifies the destination address of incoming DNS packets when it matches the dns_ip.
Specifically, it transforms the address 100.127.136.241:53 to 100.127.136.241:5053. Additionally,
it handles outgoing DNS packets returning from the NetBird resolver to DNS clients by altering the source address.
In this case, 100.127.136.241:5053 is changed to 100.127.136.241:53.

int xdp_dns_fwd(struct iphdr  *ip, struct udphdr *udp) {
    if (dns_port == 0) {
        if(!read_settings()){
            return XDP_PASS;
        }
    }

    // For DNS request processing:
    // - Incoming DNS packets going to,
    //   for example, 100.127.136.241:53 (a 'fake' address),
    //   are redirected to the NetBird resolver at 100.127.136.241:5053.
    if (udp->dest == GENERAL_DNS_PORT && ip->daddr == dns_ip) {
        udp->dest = dns_port;
        return XDP_PASS;
    }

    // For DNS response processing:
    // - The source port 5053 of the NetBird resolver's response
    //   is replaced with port 53.
    //   E.g., 100.127.136.241:5053 becomes 100.127.136.241:53.
    //   This is necessary for the client to accept the response.
    if (udp->source == dns_port && ip->saddr == dns_ip) {
        udp->source = GENERAL_DNS_PORT;
        return XDP_PASS;
    }

    return XDP_PASS;
}

Why did we attach the XDP program to the loopback interface but not the WireGuard interface wt0?
The reason is simple: the DNS resolution we are performing is local. The system understands this and optimizes the flow by using the loopback interface.
That is also why we can modify the DNS response even though XDP only works on ingress traffic.
Running tcpdump while pinging postgres.netbird.cloud demonstrates this behaviour:

misha@misha-linux:~$ sudo tcpdump -i any -n 'udp and (port 53 or port 5053)'
15:17:53.581324 lo    In  IP 100.127.136.241.59049 > 100.127.136.241.5053: UDP, length 44
15:17:53.581675 lo    In  IP 100.127.136.241.53 > 100.127.136.241.59049: 34021 0/0/0 (44)

You may have also noticed a call to the read_settings() function. We use it to make the function more flexible and pass parameters from the Go code. It helps us configure the "fake" address with the dns_ip and dns_port properties that the eBPF program uses to "catch" DNS packets.
We achieved this by using BPF maps
to pass these parameters from the userspace Go application to the eBPF program.

bool read_settings() {
    __u16 *port_value;
    __u32 *ip_value;

    // read dns ip (configured fake address)
    ip_value = bpf_map_lookup_elem(&nb_map_dns_ip, &map_key_dns_ip);
    if(!ip_value) {
        return false;
    }
    dns_ip = htonl(*ip_value);

    // read dns port (configured real active listener port. e.g. 5053)
    port_value = bpf_map_lookup_elem(&nb_map_dns_port, &map_key_dns_port);
    if (!port_value) {
        return false;
    }
    dns_port = htons(*port_value);
    return true;
}

To put it all together and use the function in Go, we've built and compiled the program with bpf2go,
which is a component of the Cilium ecosystem.
The command generates Go helper files that contain the eBPF program's bytecode.

# required packages libbpf-dev, libc6-dev-i386-amd64-cross
go run github.com/cilium/ebpf/cmd/bpf2go \
    -cc clang-14 \
    bpf src/prog.c \
    -- \
    -I /usr/x86_64-linux-gnu/include

The generated files can be used to load eBPF functions and pass parameters via a BPF map:

func (tf *GeneralManager) LoadDNSFwd(ip string, dnsPort int) error {
    log.Debugf("load ebpf DNS forwarder, watching addr: %s:53, redirect to port: %d", ip, dnsPort)
    tf.lock.Lock()
    defer tf.lock.Unlock()

    err := tf.loadXdp()
    if err != nil {
       return err
    }

    err = tf.bpfObjs.NbMapDnsIp.Put(mapKeyDNSIP, ip2int(ip))
    if err != nil {
       return err
    }

    err = tf.bpfObjs.NbMapDnsPort.Put(mapKeyDNSPort, uint16(dnsPort))
    if err != nil {
       return err
    }

    tf.setFeatureFlag(featureFlagDnsForwarder)
    err = tf.bpfObjs.NbFeatures.Put(mapKeyFeatures, tf.featureFlags)
    if err != nil {
       return err
    }
    return nil
}

You can find the complete eBPF code in our GitHub repository at the following link: GitHub Repo - netbirdio/netbird.

Conclusion

We didn't expect to use technologies like eBPF and XDP when we started working on the DNS feature, nor did we think there would be so many edge cases.
The main reason for the complexity was our integration with the kernel WireGuard, where NetBird configures the interface and steps aside.
NetBird doesn't have direct access to network packets in this mode, unlike the userspace wireguard-go implementation,
where the DNS packets can be processed directly in Go. Therefore, the userspace mode doesn't require port forwarding with XDP and eBPF.

We intentionally picked the harder path, not just because we enjoy challenges but also because the kernel WireGuard mode
offers high network performance, security, and efficiency. We are committed to bringing this value to our cloud and open-source users.

As for the userspace implementation, the NetBird DNS feature also works on Windows, macOS, and mobile phones where we used wireguard-go.
How does it work there? We will cover this in a separate article.
Meanwhile, try NetBird in the cloud with the QuickStart Guide or self-host it on your servers with the Self-hosting Quickstart Guide.