DEV Community: Sebastian Marines

[Boost]

Sebastian Marines — Wed, 16 Jul 2025 23:40:22 +0000

David Victoria for AWS Heroes

Jul 16 '25

AWS lanza su nueva capa gratuita: lo que debes saber, lo que nadie te dice y por qué es buena (aunque imperfecta)

Comments 1

9 min read

[Boost]

Sebastian Marines — Sun, 01 Dec 2024 05:49:21 +0000

The Power of Communities: Gamifying the experience of the AWS Community Day México attendee

David Victoria for AWS Community Builders ・ Dec 1

The journey of an internet packet: Exploring networks with traceroute

Sebastian Marines — Fri, 23 Aug 2024 06:00:00 +0000

We often take for granted that when we try to connect to a server, the connection will work. But that is not always the case. Sometimes there is something broken with the network that prevents us from reaching the destination. But how can we know where the problem is?

I often found myself using ping to test connectivity between two machines, and while ping is a great tool to test if a machine is reachable, it doesn't give us much information about what can be wrong.

The internet is a complex network of routers, switches, and computers, and when we try to connect to a server, our packets go through many routers before reaching the destination. If one of these routers is misconfigured or down, the packet can be dropped, and we can't reach the destination.

In this post, we will see how traceroute works, and how it can help us diagnose network problems.

The life of a packet

When you connect to a server, either a website or any other service, your computer checks if the destination IP address is in the same network. If it is, it sends the packet directly to the destination. If it is not, it sends the packet to the default gateway, which is your ISP's router that connects your network to the internet.

The router then checks if the destination IP address is in its routing table. If it is, it sends the packet directly to the destination. If it is not, it sends the packet to the next router in the path. This process repeats until the packet reaches the destination.

In the next image you can see this process. First, the computer with an IP address of 10.0.0.1 pings the computer with an IP address of 10.0.0.2 sending an ICMP packet. Since they are not in the same network, the packet is sent to the router with an IP address of 10.1.0.5. This router then forwards the packet to the router with an IP address of 10.1.0.6, which forwards the packet to the computer with an IP address of 10.1.0.7. The last router finally sends the packet to the destination computer, and the packet reaches its destination.

If we ping the destination computer we can see that it responds correctly.

$ ping 10.0.0.2
PING 10.0.0.2 (10.0.0.2): 56 data bytes
64 bytes from 10.0.0.2: icmp_seq=0 ttl=58 time=5.368 ms
64 bytes from 10.0.0.2: icmp_seq=1 ttl=58 time=4.325 ms
64 bytes from 10.0.0.2: icmp_seq=2 ttl=58 time=4.225 ms
64 bytes from 10.0.0.2: icmp_seq=3 ttl=58 time=4.479 ms

What happens when there is a problem?

But now, what happens if there is a problem in the path? For example, what happens if one of the routers is misconfigured and drops the packet? Or what happens if the destination computer is down? How can we know where the packet is being dropped?

Take a look at the following example, where the router 10.1.0.6 can't communicate with the router 10.1.0.7.

If we ping the destination computer we can see that we can't reach the target, but we don't know where the problem is.

$ ping 10.0.0.2
PING 10.0.0.2 (10.0.0.2) 56(84) bytes of data.
^C

--- 10.0.0.2 ping statistics ---
4 packets transmitted, 0 received, +4 errors, 100% packet loss, time 3031ms

This is where traceroute comes in handy. traceroute is a network diagnostic tool that shows the path that a packet takes from the source to the destination. It allows us to see every router in the path and the response time of each one.

$ traceroute 10.0.0.2
traceroute to 10.0.0.2 (10.0.0.2), 30 hops max, 60 byte packets
 1  10.1.0.5  1.123 ms  0.912 ms  1.145 ms
 2  10.1.0.6  2.145 ms  2.023 ms  2.311 ms
 3  * * *
 4  * * *
 5  * * *

In this example we can see that routers 10.1.0.5 and 10.1.0.6 are working correctly, but there is a problem with the next router in the path. You can see that we first reach the router 10.1.0.5 and the response time is around 1ms. Then we reach the router 10.1.0.6, with a response time of around 2ms. But then, the path is broken, and we can't reach the destination computer.

How does `traceroute` work?

To understand how traceroute works, we first need to understand the fields in an IP header.

There are several fields in the IP header, but we will focus on 3 of them:

Source IP address: The IP address of the computer that sends the packet.
Destination IP address: The IP address of the computer that receives the packet.
Time to live (TTL): The maximum number of routers that the packet can pass through.

The source and destination IP addresses are self-explanatory, but the TTL field is an interesting one. This field is used to prevent packets from looping indefinitely in a network.

When a packet is sent, the TTL field is set to a value of 64 (at least in Linux and macOS). When the packet reaches a router, the router decrements the TTL field by 1. If the TTL field reaches 0, the router drops the packet and sends an Time Exceeded Message back to the source computer. But if the TTL field is greater than 0, the router forwards the packet to the next router in the path.

This is how traceroute is able to show the path that a packet takes from the source to the destination. It sends an ICMP packet with a TTL of 1, then a packet with a TTL of 2, then a packet with a TTL of 3, and so on. This way, traceroute can see the response of each router in the path.

What is ICMP?

Up to this point, we have mentioned ICMP a couple of times, but what is ICMP?

ICMP stands for Internet Control Message Protocol. It is a protocol used by network devices to diagnose network communication issues¹.

This protocol is used by commands like ping and traceroute to test network connectivity.

The ICMP standard² defines several message types. The most common ones are:

Echo Request: Used by the ping command to test if a computer is reachable.
Echo Reply: Sent by a computer in response to an Echo Request.
Time Exceeded Message: Sent by a router when the TTL field of a packet reaches 0.
Destination Unreachable Message: Sent by a router when the destination computer is unreachable.
Redirect Message: Sent by a router to inform a computer that there is a better route to a destination.
Parameter Problem Message: Sent by a router when a packet has an incorrect header.
Source Quench Message: Sent by a router to inform a computer that it is sending too many packets.

Visualizing `traceroute`

Let's run traceroute to the destination computer 10.0.0.2 and see what happens in each step.

$ traceroute 10.0.0.2
traceroute to 10.0.0.2 (10.0.0.2), 64 hops max, 40 byte packets
...

First, traceroute sends an ICMP packet with a TTL of 1. The packet reaches the first router in the path, which decrements the TTL field by 1. Since the TTL field is now 0, the router drops the packet and sends an Time Exceeded Message back to the source computer.

$ traceroute 10.0.0.2
traceroute to 10.0.0.2 (10.0.0.2), 64 hops max, 40 byte packets
 1 10.1.0.5 (10.1.0.5)  1.123 ms  0.912 ms  1.145 ms
...

You can see that we have 3 response times for the first router. This is because traceroute sends 3 packets to each router.

Next, traceroute sends an ICMP packet with a TTL of 2. The packet reaches the first router in the path, which decrements the TTL field by 1. The TTL field is now 1, so the router forwards the packet to the next router in the path.

The packet reaches the second router in the path, which decrements the TTL field by 1. Since the TTL field is now 0, the router drops the packet and sends an Time Exceeded Message back to the source computer.

$ traceroute 10.0.0.2
traceroute to 10.0.0.2 (10.0.0.2), 64 hops max, 40 byte packets
 1 10.1.0.5 (10.1.0.5)  1.123 ms  0.912 ms  1.145 ms
 2 10.1.0.6 (10.1.0.6)  2.145 ms  2.023 ms  2.311 ms
...

Now, traceroute sends an ICMP packet with a TTL of 3. The packet reaches the first router in the path, which decrements the TTL field by 1. The TTL field is now 2, so the router forwards the packet to the next router in the path.

When the packet reaches the last router in the path, the router decrements the TTL field by 1. Since the TTL field is now 0, the router drops the packet and sends an Time Exceeded Message back to the source computer.

$ traceroute
traceroute to 10.0.0.2 (10.0.0.2), 64 hops max, 40 byte packets
 1 10.1.0.5 (10.1.0.5)  1.123 ms  0.912 ms  1.145 ms
 2 10.1.0.6 (10.1.0.6)  2.145 ms  2.023 ms  2.311 ms
 3 10.1.0.7 (10.1.0.7)  3.145 ms  3.023 ms  3.311 ms
...

Finally, traceroute sends an ICMP packet with a TTL of 4. The packet reaches the first router in the path, which decrements the TTL field by 1. Then each router in the path decrements the TTL field by 1 until the packet reaches the destination computer.

The destination computer receives the packet and sends an ICMP Echo Reply back to the source computer.

$ traceroute 10.0.0.2
traceroute to 10.0.0.2 (10.0.0.2), 64 hops max, 40 byte packets
 1 10.1.0.5 (10.1.0.5)  1.123 ms  0.912 ms  1.145 ms
 2 10.1.0.6 (10.1.0.6)  2.145 ms  2.023 ms  2.311 ms
 3 10.1.0.7 (10.1.0.7)  3.145 ms  3.023 ms  3.311 ms
 4 10.0.0.2 (10.0.0.2)  4.145 ms  4.023 ms  4.311 ms

Conclusion

Traceroute is a powerful diagnostic tool that helps us understand the path our data takes through the internet. It provides valuable insights into network topology and potential issues along the way, and allows us to:

Identify where packets are being dropped or delayed
Discover unexpected routing paths
Detect misconfigured or malfunctioning routers
Estimate network latency between hops

While ping can tell us if a destination is reachable, traceroute shows us the entire journey, making it an essential tool in any network administrator's or curious user's toolkit.

As we've seen, the internet is a complex web of interconnected devices, and traceroute helps simplify this complexity. Next time you encounter a network issue, remember to use traceroute, it might help you find the problem and save valuable troubleshooting time.

Connecting to the DN42 decentralized network

Sebastian Marines — Sat, 10 Aug 2024 17:50:00 +0000

I have always been interested in how the internet works, it has marvelled me how we can communicate with people from across the globe in milliseconds, but I never really understood how it worked. But a few months ago I stumbled upon this article that explained how to build your own anycast network, and introduced me to some concepts that I had never heard of before, like BGP and ASNs.

The DN42 network is a decentralized network that uses VPNs and BGP to create an "internet" over the internet. This means that anyone who is interested on learning the technologies that ISPs use to create the internet can join and learn how it works without the need of expensive hardware or messing up with the real internet ¹.

Preparation

Before you can connect to the DN42 network you need to register an Autonomous System Number (ASN), network prefixes, and routes. This process is done by creating a pull request to the DN42 registry, where you specify the ASN you want to use, the network prefixes you want to announce, and the routes you want to receive. The getting started guide on the DN42 website explains how to do this in detail.

The ASN is a unique number that identifies your network on the internet, each ISP in the real internet has one or more ASNs that they use to announce their network prefixes to the rest of the internet, and we will use it to do the same on the DN42 network.

My ASN is 4242422799 and I have the following network prefixes:

IPv4: 172.20.196.64/27
IPv6: fd44:665c:b6b3::/48

I will be using these prefixes to announce my network to the rest of the DN42 network, and I will be receiving the routes of the other members of the network to be able to communicate with them.

Requirements

Software

We need the following software to connect to DN42:

A VPN client (I will be using WireGuard)
A BGP daemon (I will be using BIRD)
Something to manage routes (I will be using IPTables)

There are other alternatives to the software I will be using, like OpenVPN for the VPN client or Quagga for the BGP daemon, but I chose these because they are the ones that I found the easiest to use and the ones that most people in the DN42 community use.

Hardware

For the hardware requirements, you need a router that supports the software you will be using, or any computer that can act as a router. I chose to use a small VPS from IONOS with 1 vCPU, 1GB of RAM, and 10GB of storage, but you can use any computer that you have lying around.

Configuration

Creating your WireGuard keys

To peer with other members you need to create a tunnel with them, and to do that you need to create a pair of public and private keys needed to authenticate the connection. You can do this by running the following commands:

# Generate the private key
$ wg genkey > privatekey
# Generate the public key
$ wg pubkey < privatekey > publickey

We will need the public key for the following steps, so keep it handy.

Choosing a peer

The first thing you need to do in order to connect to the DN42 network is to find a peer that is willing to peer with you. You can use a tool like PingFinder to find peers that are close to you geographically, or you can ask in the DN42 IRC channel for someone to peer with you.

Other networks like Kioubit have an automated peering system that allows you to peer with other members of the network, by filling some information about your network in their website. You can find a list of systems with automated peering here.

I chose to peer with Kioubit specifically because of their automated peering system and because it is one of the biggest networks in the DN42 community. Once you authenticate to the Kioubit website and choose a node to peer with, we need to fill in the information about our network in the form.

First we need to fill our WireGuard public key that we generated previously. Then, for the Tunnel IPv4 address we will use the first non-zero IP address of our network prefix, in my case 172.20.196.65. And finally, for the Tunnel IPv6 address we will generate a link-local address ² using the fe80::/10 prefix, in my case fe80::fe80::abcd.

Once we fill in the information and submit the form, the information we need to connect to the Kioubit node will be displayed on the screen.

Creating the WireGuard tunnel

Now that we have the information we need to connect to the Kioubit node, we can create the WireGuard tunnel. We will create a configuration file with the information we got from the Kioubit website and add it to the WireGuard interface.

# /etc/wireguard/kioubit.conf
[Interface]
PrivateKey = <YOUR PRIVATE KEY>

[Peer]
PublicKey = 6Cylr9h1xFduAO+5nyXhFI1XJ0+Sw9jCpCDvcqErF1s=
Endpoint = us2.g-load.eu:22799
AllowedIPs = 0.0.0.0/0,::/0

With the configuration file created, we can add it to the WireGuard interface and enable it.

# Create the WireGuard interface
$ ip link add dev "kioubit" type wireguard
# Add the configuration file to the interface
$ wg setconf "kioubit" /etc/wireguard/kioubit.conf
# Add our link-local IPv6 address to the interface
$ ip addr add fe80::abcd/64 dev "kioubit"
# Peer with the Kioubit node
# The first IP is our IP and the second IP is the Kioubit node IP
$ ip addr add 172.20.196.65/32 peer 172.20.53.98/32 dev "kioubit"
# Enable the interface
$ ip link set "kioubit" up

Configuring BIRD

Now that we have the WireGuard tunnel up and running, we need to configure BIRD to announce our network prefixes to the rest of the DN42 network and to receive the routes of the other members.

First, we need to create the configuration file for bird.

# /etc/bird.conf

################################################
#               Variable header                #
################################################

define OWNAS =  4242422799; # Your ASN
define OWNIP =  172.20.196.65; # The first IP of your IPv4 prefix
define OWNIPv6 = fd44:665c:b6b3::1; # The first IP of your IPv6 prefix
define OWNNET = 172.20.196.64/27; # Your IPv4 prefix
define OWNNETv6 = fd44:665c:b6b3::/48; # Your IPv6 prefix
define OWNNETSET = [172.20.196.64/27+];
define OWNNETSETv6 = [fd44:665c:b6b3::/48+];

################################################
#                 Header end                   #
################################################

router id OWNIP;

protocol device {
    scan time 10;
}

/*
 *  Utility functions
 */

function is_self_net() {
  return net ~ OWNNETSET;
}

function is_self_net_v6() {
  return net ~ OWNNETSETv6;
}

function is_valid_network() {
  return net ~ [
    172.20.0.0/14{21,29}, # dn42
    172.20.0.0/24{28,32}, # dn42 Anycast
    172.21.0.0/24{28,32}, # dn42 Anycast
    172.22.0.0/24{28,32}, # dn42 Anycast
    172.23.0.0/24{28,32}, # dn42 Anycast
    172.31.0.0/16+,       # ChaosVPN
    10.100.0.0/14+,       # ChaosVPN
    10.127.0.0/16{16,32}, # neonetwork
    10.0.0.0/8{15,24}     # Freifunk.net
  ];
}

roa4 table dn42_roa;
roa6 table dn42_roa_v6;

protocol static {
    roa4 { table dn42_roa; };
    include "/etc/bird/roa_dn42.conf";
};

protocol static {
    roa6 { table dn42_roa_v6; };
    include "/etc/bird/roa_dn42_v6.conf";
};

function is_valid_network_v6() {
  return net ~ [
    fd00::/8{44,64} # ULA address space as per RFC 4193
  ];
}

protocol kernel {
    scan time 20;

    ipv6 {
        import none;
        export filter {
            if source = RTS_STATIC then reject;
            krt_prefsrc = OWNIPv6;
            accept;
        };
    };
};

protocol kernel {
    scan time 20;

    ipv4 {
        import none;
        export filter {
            if source = RTS_STATIC then reject;
            krt_prefsrc = OWNIP;
            accept;
        };
    };
}

protocol static {
    route OWNNET reject;

    ipv4 {
        import all;
        export none;
    };
}

protocol static {
    route OWNNETv6 reject;

    ipv6 {
        import all;
        export none;
    };
}

template bgp dnpeers {
    local as OWNAS;
    path metric 1;

    ipv4 {
        import filter {
          if is_valid_network() && !is_self_net() then {
            if (roa_check(dn42_roa, net, bgp_path.last) != ROA_VALID) then {
              # Reject when unknown or invalid according to ROA
              print "[dn42] ROA check failed for ", net, " ASN ", bgp_path.last;
              reject;
            } else accept;
          } else reject;
        };

        export filter { if is_valid_network() && source ~ [RTS_STATIC, RTS_BGP] then accept; else reject; };
        import limit 9000 action block;
    };

    ipv6 {   
        import filter {
          if is_valid_network_v6() && !is_self_net_v6() then {
            if (roa_check(dn42_roa_v6, net, bgp_path.last) != ROA_VALID) then {
              # Reject when unknown or invalid according to ROA
              print "[dn42] ROA check failed for ", net, " ASN ", bgp_path.last;
              reject;
            } else accept;
          } else reject;
        };
        export filter { if is_valid_network_v6() && source ~ [RTS_STATIC, RTS_BGP] then accept; else reject; };
        import limit 9000 action block; 
    };
}


include "/etc/bird/peers/*";

Next, we need to download the ROA files that will be used to validate the routes we receive from the other members of the DN42 network.

$ wget https://dn42.burble.com/roa/dn42_roa_bird2_4.conf -O /etc/bird/roa_dn42.conf
$ wget https://dn42.burble.com/roa/dn42_roa_bird2_6.conf -O /etc/bird/roa_dn42_v6.conf

Finally, we need to create the configuration file for the Kioubit peer.

# /etc/bird/peers/kioubit.conf
protocol bgp kioubit from dnpeers {
    # The IP address of the Kioubit node
    neighbor 172.20.53.98 as 4242423914 ;
}

protocol bgp kioubit_v6 from dnpeers {
    # Link-local address of the Kioubit node
    neighbor fe80::ade0%kioubit as kioubit;
}

With the configuration files created, we can start the BIRD daemon.

$ systemctl start bird

Configuring forwarding

To be able to route the traffic between the WireGuard interface and the rest of the network, we need to enable IP forwarding in the kernel.

$ sysctl -w net.ipv4.ip_forward=1

Testing

Now that we have everything set up, we can test the connection by pinging the Kioubit node.

$ ping 172.20.53.98
PING 172.20.53.98 (172.20.53.98) 56(84) bytes of data.
64 bytes from 172.20.53.98: icmp_seq=1 ttl=64 time=3.07 ms
64 bytes from 172.20.53.98: icmp_seq=2 ttl=64 time=3.05 ms
64 bytes from 172.20.53.98: icmp_seq=3 ttl=64 time=3.17 ms
64 bytes from 172.20.53.98: icmp_seq=4 ttl=64 time=3.07 ms
64 bytes from 172.20.53.98: icmp_seq=5 ttl=64 time=3.18 ms
^C
--- 172.20.53.98 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4005ms
rtt min/avg/max/mdev = 3.048/3.108/3.182/0.055 ms

We can also try resolving a domain name using the DNS anycast service

$ dig @172.20.0.53 burble.dn42
; <<>> DiG 9.16.23-RH <<>> @172.20.0.53 burble.dn42
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 56811
;; flags: qr rd ra ad; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;burble.dn42.           IN  A

;; ANSWER SECTION:
burble.dn42.        3600    IN  A   172.20.129.3

;; Query time: 27 msec
;; SERVER: 172.20.0.53#53(172.20.0.53)
;; WHEN: Sun Aug 11 11:32:55 UTC 2024
;; MSG SIZE  rcvd: 56

Conclusion

Connecting to the DN42 network was a great learning experience for me, I learned a lot about the technologies that make the internet work. I would recommend anyone who is interested in networking to join the DN42 network and learn how the internet works from the inside.

There is still a lot to learn and explore in the DN42 network, like setting up a DNS server, a web server, or even a mail server. I will be exploring these topics in future posts, so stay tuned!

Understanding Linux's File Descriptors: A Deep Dive Into '2>&1' and Redirection

Sebastian Marines — Thu, 25 Jan 2024 17:11:22 +0000

You have probably seen this syntax before:

$ command > file 2>&1

This redirects the standard output and standard error of command to file. But what does it mean? What are file descriptors? What is standard output? Let's find out.

File descriptors

A file descriptor is the Unix abstraction for an open input/output stream: a file, a network connection, a pipe (a communication channel between processes), a terminal, etc.¹

Every process normally has 3 file descriptors that are open by default and are inherited from the parent process (usually the shell)

Integer value	Name	`<unistd.h>` symbolic constant	`<stdio.h>` file stream
0	Standard input	`STDIN_FILENO`	`stdin`
1	Standard output	`STDOUT_FILENO`	`stdout`
2	Standard error	`STDERR_FILENO`	`stderr`

Source: File Descriptor - Wikipedia

When a process opens a file (remember that everything in Unix is a file, including devices like the terminal, sockets, pipes, etc.), the kernel assigns a file descriptor to it. This file descriptor is an integer that uniquely identifies the file for the process.

Internally, the kernel keeps a table of file descriptors for each process. This table is called the file descriptor table. Each entry in the table contains information about the file, such as the file offset, the file status flags, etc.

When a process opens a file, the kernel returns the lowest available file descriptor. This means that the first file opened by a process will have file descriptor 3, the second file will have file descriptor 4, and so on.

Take a look at the following C program:

#include <stdio.h>
int main()
{
    fprintf(stdout, "I'm writing to stdout\n");
    fprintf(stderr, "I'm writing to stderr\n");
}

The program prints two lines, one to standard output and one to standard error. Let's compile and run it:

$ gcc -o print-fd print-fd.c
$ ./print-fd
I'm writing to stdout
I'm writing to stderr

Both lines are printed to the terminal. But why?

Inspecting file descriptors for a process

To inspect the file descriptors for a process, we can use the /proc filesystem. This filesystem is a virtual filesystem that provides information about the system and processes running on it. It is usually mounted at /proc.

It contains a directory for each process running on the system. The name of the directory is the process ID. For example, the directory for the current process is /proc/self.

Each directory contains a lot of information about the process, but we are interested in the fd directory. This directory contains a symlink for each file descriptor open by the process. The name of the symlink is the file descriptor number and the target is the file the descriptor is pointing to.

$ ls -l /proc/self/fd
total 0
lrwx------. 1 sebastian sebastian 64 Jan 22 19:28 0 -> /dev/pts/0
lrwx------. 1 sebastian sebastian 64 Jan 22 19:28 1 -> /dev/pts/0
lrwx------. 1 sebastian sebastian 64 Jan 22 19:28 2 -> /dev/pts/0
lr-x------. 1 sebastian sebastian 64 Jan 22 19:28 3 -> /proc/2645/fd

As you can see, bash has 3 file descriptors open by default: 0, 1, and 2. All of them are pointing to the same file: /dev/pts/0. This is the terminal where the process is running.

What does 2>&1 mean?

Now that we know what file descriptors are, we can understand what the syntax 2>&1 means.

$ ./print-fd > file.txt 2>&1

This redirects the standard output of ./print-fd to file.txt and redirects the standard error (file descriptor 2) of ./print-fd to the same place as standard output (file descriptor 1).

Let's see other examples:

$ # Redirect standard error to /dev/null
$ ./print-fd 2> /dev/null
I'm writing to stdout

$ # Redirect standard output to /dev/null and standard error to the current terminal
$ ./print-fd > /dev/null 2> $(tty)
I'm writing to stderr

How redirections work

With this information, we can understand how redirections work. Let's take a look at the following command bash script:

#!/bin/bash
echo "hello" > /tmp/1234

This command redirects the standard output of echo to the file /tmp/1234. Let's see what happens when we run it and inspect the syscalls using strace²:

strace -f -e trace=write,dup2,read,openat ./test.sh

Note: strace is a tool that allows us to inspect the syscalls a process is making. It is very useful for debugging and understanding how programs work. We are limiting the output to only show the syscalls we are interested in.

...
read(3, "# test.sh\n\n#!/bin/bash\necho \"hel"..., 80) = 48
dup2(3, 255)                            = 255
read(255, "# test.sh\n\n#!/bin/bash\necho \"hel"..., 48) = 48
openat(AT_FDCWD, "/tmp/1234", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 3
dup2(3, 1)                              = 1
write(1, "hello\n", 6)                  = 6
...
+++ exited with 0 +++

The first line of the output is the read syscall. It is using file descriptor 3 to read the script and trying to read 80 characters, but only reads 48. This is because the script is only 48 characters long.
The next 2 lines are the dup2 syscall. It is duplicating file descriptor 3 to file descriptor 255. This is because the next syscall is also using file descriptor 3, so we need to duplicate it to another file descriptor to avoid closing it.
Next, is the openat syscall. It is opening the file /tmp/1234 with the flags O_WRONLY|O_CREAT|O_TRUNC. This means that the file will be opened in write-only mode, it will be created if it doesn't exist, and it will be truncated to 0 length if it exists. It returns file descriptor 3, that was the file descriptor that was freed by the previous dup2 syscall.
The next dup2 syscall is duplicating file descriptor 3 to file descriptor 1. This is because we want to redirect the standard output of echo to the file /tmp/1234 (Remember that standard output is file descriptor 1).
The next syscall is the write syscall. It is writing the string hello\n to file descriptor 1, which is the file /tmp/1234.

Implementing redirections from scratch

Understanding the process bash follows to implement redirections is very useful, but what if we want to implement redirections from scratch? How can we do it?

To do this, we can follow this process:

Fork the process
In the child process, open the files to redirect to using open
Using the dup2 syscall, redirect standard output and standard error to the files
Use the execvp syscall to replace the current process with the underlying program

Let's see how this works in practice. The following C program redirects the standard output and standard error of the underlying program to the files /out.log and /error.log respectively.

#include <stdio.h>
#include <fcntl.h>
#include <unistd.h>
#include <sys/types.h>
#include <sys/wait.h>

int main(int argc, char * argv[]) {
  pid_t pid;
  int status;

  // Fork the process
  pid = fork();
  if (pid == -1) {
    perror("fork");
    return 1;
  }

  if (pid == 0) {
    // We are in the child process

    // Open the error file, with write-only permissions and create it if it doesn't exist
    int newerr = open("/error.log", O_WRONLY | O_CREAT, 0666);
    if (newerr == -1) {
      perror("open");
      return 1;
    }

    // Open the output file, with write-only permissions and create it if it doesn't exist
    int newout = open("/out.log", O_WRONLY | O_CREAT, 0666);
    if (newout == -1) {
      perror("open");
      return 1;
    }

    // Make stderr file descriptor point to the error file
    if (dup2(newerr, STDERR_FILENO) == -1) {
      perror("dup2");
      return 1;
    }

    // Make stdout file descriptor point to the output file
    if (dup2(newout, STDOUT_FILENO) == -1) {
      perror("dup2");
      return 1;
    }

    // Replace the current process with the print-fd program
    // Notice that we are replacing the process of the forked child, not the original process
    char * newargv[] = {
      "/print-fd",
      NULL
    };
    execvp(newargv[0], newargv);

    // execvp only returns if there is an error
    perror("execvp");
    return 1;
  } else {
    // We are in the parent process
    // Wait for the child to finish
    if (waitpid(pid, & status, 0) == -1) {
      perror("waitpid");
      return 1;
    }

    // Check child's exit status
    if (WIFEXITED(status)) {
      printf("Child exited with status %d\n", WEXITSTATUS(status));
    } else {
      printf("Child did not exit cleanly\n");
    }
  }

  return 0;
}

Now, let's compile and run it:

$ gcc -o redirect redirect.c
$ ./redirect
Child exited with status 0
$ $ tail -n +1 /*.log
==> /error.log <==
I'm writing to stderr

==> /out.log <==
I'm writing to stdout

Conclusion

In this post, we learned what file descriptors are, how they work, and how to implement redirections from scratch. I hope you found this post useful and learned something new.

File Descriptors - Harvard CS61 ↩
strace ↩

Deploying Go to AWS Lambda with GitHub Actions and Terraform

Sebastian Marines — Thu, 13 Jul 2023 03:03:28 +0000

Introduction

Deploying serverless applications has become increasingly popular due to its scalability, cost-efficiency, and ease of management. Amazon Web Services (AWS) Lambda provides a powerful serverless computing platform, while Go (Golang) offers a performant and statically typed programming language.

In this article, we will explore how to deploy Go-based applications to AWS Lambda using the seamless integration of GitHub Actions and Terraform. By combining these tools, developers can automate the deployment process, streamline version control, and ensure consistent infrastructure as code practices for their serverless projects.

Prerequisites

Before we begin, you will need to have the following prerequisites:

An AWS account
A GitHub account
AWS CLI installed and configured with your AWS credentials (see the AWS documentation for instructions)
Go installed (see the Go documentation for instructions)
Terraform installed (see the Terraform documentation for instructions)

Step 1: Create a GitHub repository

First of all, create a GitHub repository for your Go application.

Go to GitHub and click on "New repository". Enter a name for your repository and click on "Create repository".

Step 2: Create a Go application

In an empty directory, create a Go application:

go mod init go_lambda

This will create a go.mod file that will be used to manage dependencies for your Go application.

Next, initialize the Git repository:

git init

Next, create a .gitignore file:

# .gitignore
**/.terraform/*
*.tfstate
bootstrap
lambda-handler.zip

Lastly, set the remote origin:

git remote add origin <YOUR_REPOSITORY_URL>

Step 3: Create a main.go file

Next, create the main.go file:

// main.go
package main

import (
 "github.com/aws/aws-lambda-go/events"
 "github.com/aws/aws-lambda-go/lambda"
)

func hello() (events.APIGatewayProxyResponse, error) {
 return events.APIGatewayProxyResponse{
  Body:       "Hello World!",
  StatusCode: 200,
 }, nil
}

func main() {
 // Make the handler available for Remote Procedure Call by AWS Lambda
 lambda.Start(hello)
}

This file contains the minimal code required to create a Go application that can be deployed to AWS Lambda. It imports the events and lambda packages from the aws-lambda-go library and defines a hello function that returns a response that our API Gateway will return to the client. The main function is the entry point for our application and calls the hello function. The lambda.Start function starts the AWS Lambda handler¹.

Step 4: Install dependencies

Next, install the dependencies for your Go application:

go get github.com/aws/aws-lambda-go/events
go get github.com/aws/aws-lambda-go/lambda

This will install aws-lambda-go/events, which contains the APIGatewayProxyResponse struct, and aws-lambda-go/lambda, which contains the lambda.Start function.

Step 5: Create the remote backend

We will use S3 as our remote backend for Terraform. This is necessary because we will be using GitHub Actions to deploy our code and we need to store the Terraform state in a remote location.

NOTE We will not be creating a DynamoDB table for state locking to keep things simple, but you should consider doing this if you are deploying to production. You can read more about this in the Terraform documentation.

To create the S3 bucket for our remote backend, we will use the following command:

aws s3api create-bucket --bucket <YOUR_BUCKET_NAME> --region us-east-1

Alternatively, you can create the bucket using the AWS Console following the instructions in the AWS documentation.

Step 6: Creating the Terraform configuration

Now that we have created our Go application and installed the dependencies, we can create the Terraform configuration. We will use the following Terraform configuration to deploy our Go application to AWS Lambda:

# providers.tf
terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "5.7.0"
    }
  }
}

# backend.tf
terraform {
  backend "s3" {
    # Replace this with your bucket name!
    bucket = "<YOUR_BUCKET_NAME>"
    key    = "go-lambda-test.tfstate"
    region = "us-east-1"
  }
}

# main.tf
resource "aws_lambda_function" "go_function" {
  filename      = "lambda-handler.zip"
  function_name = "go-lambda-test"
  handler       = "bootstrap"
  role = aws_iam_role.iam_for_lambda.arn

  source_code_hash = filebase64sha256("lambda-handler.zip")

  runtime = "go1.x"
}

data "aws_iam_policy_document" "assume_role" {
  statement {
    effect = "Allow"

    principals {
      type        = "Service"
      identifiers = ["lambda.amazonaws.com"]
    }

    actions = ["sts:AssumeRole"]
  }
}

resource "aws_iam_role" "iam_for_lambda" {
  name               = "iam_for_lambda"
  assume_role_policy = data.aws_iam_policy_document.assume_role.json
}

# API Gateway

resource "aws_api_gateway_rest_api" "go_api" {
  name        = "go_api"
  description = "This is my API for demonstration purposes"
}

resource "aws_api_gateway_resource" "go_api_resource" {
  rest_api_id = aws_api_gateway_rest_api.go_api.id
  parent_id   = aws_api_gateway_rest_api.go_api.root_resource_id
  path_part   = "test"
}

resource "aws_api_gateway_method" "go_api_method" {
  rest_api_id   = aws_api_gateway_rest_api.go_api.id
  resource_id   = aws_api_gateway_resource.go_api_resource.id
  http_method   = "GET"
  authorization = "NONE"
}

resource "aws_api_gateway_integration" "go_api_integration" {
  rest_api_id = aws_api_gateway_rest_api.go_api.id
  resource_id = aws_api_gateway_resource.go_api_resource.id
  http_method = aws_api_gateway_method.go_api_method.http_method

  integration_http_method = "POST"
  type                    = "AWS_PROXY"
  uri                     = aws_lambda_function.go_function.invoke_arn
}

resource "aws_api_gateway_method" "proxy_root" {
  rest_api_id   = aws_api_gateway_rest_api.go_api.id
  resource_id   = aws_api_gateway_rest_api.go_api.root_resource_id
  http_method   = "ANY"
  authorization = "NONE"
}

resource "aws_api_gateway_integration" "proxy_root_integration" {
  rest_api_id = aws_api_gateway_rest_api.go_api.id
  resource_id = aws_api_gateway_rest_api.go_api.root_resource_id
  http_method = aws_api_gateway_method.proxy_root.http_method

  integration_http_method = "POST"
  type                    = "AWS_PROXY"
  uri                     = aws_lambda_function.go_function.invoke_arn
}

resource "aws_api_gateway_deployment" "go_api_deployment" {
  depends_on = [
    aws_api_gateway_integration.go_api_integration,
    aws_api_gateway_integration.proxy_root_integration,
  ]

  rest_api_id = aws_api_gateway_rest_api.go_api.id
  stage_name  = "test"
}

resource "aws_lambda_permission" "apigw" {
  statement_id  = "AllowAPIGatewayInvoke"
  action        = "lambda:InvokeFunction"
  function_name = aws_lambda_function.go_function.function_name
  principal     = "apigateway.amazonaws.com"
  source_arn    = "${aws_api_gateway_rest_api.go_api.execution_arn}/*/*"
}

Step 7: Configure AWS credentials for GitHub Actions

To configure AWS credentials for GitHub Actions, we will use OIDC federation. This is a secure way to authenticate with AWS without having to store your AWS credentials in your GitHub repository.

We will need to follow the following steps to configure AWS credentials for GitHub Actions:

Open the AWS Console and navigate to the IAM service.
Click on "Identity providers" in the left navigation bar.
Click on "Add provider".
Select "OpenID Connect" as the provider type.
For "Provider URL", enter https://token.actions.githubusercontent.com².
Click on "Get thumbprint".
For "Audience", enter sts.amazonaws.com².
Click on "Add provider".

Next, we will need to create an IAM role for GitHub Actions. We will need to follow the following steps to create an IAM role for GitHub Actions:

Click on the provider you just created. It should be called "token.actions.githubusercontent.com".
Click on "Assign role".
Select "Create a new role" and click on "Next".
For "Identity provider", select "token.actions.githubusercontent.com".
For Audience, enter sts.amazonaws.com.
Click on "Next: Permissions".
Select "AmazonS3FullAccess", "AWSLambdaFullAccess", "IAMFullAccess", and "AmazonAPIGatewayAdministrator" and click on "Next: Tags".
Click on "Next: Review".
For "Role name", enter github-actions and click on "Create role".
Search for the role you just created and click on it.
Copy the "ARN" (It should look something like arn:aws:iam::000000000000:role/github-actions). You will need this in the next step.

NOTE: You should consider using a more restrictive IAM policy for your GitHub Actions IAM role.

Step 8: Create the GitHub Actions workflow

Next, we will create the GitHub Actions workflow. We will use the following workflow to deploy our Go application to AWS Lambda:

Do not forget to replace <IAM_ROLE> with the IAM role ARN you created in the previous step.

# .github/workflows/deploy.yml
name: 'Deploy'

on:
  push:
    branches: [ "main" ]
  pull_request:
  workflow_dispatch:

permissions:
  id-token: write
  contents: read

jobs:
  terraform:
    name: 'Terraform'
    runs-on: ubuntu-latest
    environment: production

    defaults:
      run:
        shell: bash

    steps:
    # Checkout the repository to the GitHub Actions runner
    - name: Checkout
      uses: actions/checkout@v3

    # Configure AWS Credentials
    # You will need to replace <IAM_ROLE> with the IAM role ARN you created in the previous step
    - name: Configure AWS Credentials
      uses: aws-actions/configure-aws-credentials@v2
      with:
        role-to-assume: <IAM_ROLE>
        aws-region: us-east-1

    - name: Setup Terraform
      uses: hashicorp/setup-terraform@v1

    - name: Setup Go environment
      uses: actions/setup-go@v4.0.1

    # This step builds the Go application and creates a zip file containing the binary
    # It is important to note that the binary must be named "bootstrap"
    - name: Build Go application
      run: |
        GOOS=linux GOARCH=amd64 CGO_ENABLED=0 go build -o bootstrap main.go
        zip lambda-handler.zip bootstrap

    # Initialize a new or existing Terraform working directory by creating initial files, loading any remote state, downloading modules, etc.
    - name: Terraform Init
      run: terraform init

    - name: Terraform Format
      run: terraform fmt -check

    - name: Terraform Plan
      run: terraform plan -input=false

    - name: Output ref and event_name
      run: |
        echo ${{github.ref}}
        echo ${{github.event_name}}

      # On push to "main", build or change infrastructure according to Terraform configuration files
    - name: Terraform Apply
      if: github.ref == 'refs/heads/main' && github.event_name == 'push'
      run: terraform apply -auto-approve -input=false

    - name: Output API Gateway invocation URL
        if: github.ref == 'refs/heads/main' && github.event_name == 'push'
        run: |
          terraform output api_endpoint

Step 9: Deploy the Go application to AWS Lambda

Now that we have created the GitHub Actions workflow, we can push our code to GitHub and let GitHub Actions deploy our Go application to AWS.

git add .
git commit -m "Deploy Go application to AWS Lambda"
git push --set-upstream origin main

You can view the GitHub Actions workflow by going to the "Actions" tab in your GitHub repository.

Step 10: Test the API

Now that we have deployed our Go application to AWS Lambda, we can get the API Gateway invocation URL by going to the "Actions" tab in your GitHub repository and clicking on the latest workflow run. Then seeing the output of the "Output API Gateway invocation URL" step.

If we make a request to the API Gateway invocation URL, we should see the following response:

Conclusion

In this article, we explored how to deploy Go-based applications to AWS Lambda using GitHub Actions and Terraform. By combining these tools, developers can automate the deployment process, streamline version control, and ensure consistent infrastructure as code practices for their serverless projects.

You can find the source code for this article in the GitHub repository.

DEV Community: Sebastian Marines

[Boost]

AWS lanza su nueva capa gratuita: lo que debes saber, lo que nadie te dice y por qué es buena (aunque imperfecta)

[Boost]

The Power of Communities: Gamifying the experience of the AWS Community Day México attendee

David Victoria for AWS Community Builders ・ Dec 1

The journey of an internet packet: Exploring networks with traceroute

The life of a packet

What happens when there is a problem?

How does traceroute work?

What is ICMP?

Visualizing traceroute

Conclusion

Connecting to the DN42 decentralized network

Preparation

Requirements

Software

Hardware

Configuration

Creating your WireGuard keys

Choosing a peer

Creating the WireGuard tunnel

Configuring BIRD

Configuring forwarding

Testing

Conclusion

Understanding Linux's File Descriptors: A Deep Dive Into '2>&1' and Redirection

File descriptors

Inspecting file descriptors for a process

What does 2>&1 mean?

How redirections work

Implementing redirections from scratch

Conclusion

Deploying Go to AWS Lambda with GitHub Actions and Terraform

Introduction

Prerequisites

Step 1: Create a GitHub repository

Step 2: Create a Go application

Step 3: Create a main.go file

Step 4: Install dependencies

Step 5: Create the remote backend

Step 6: Creating the Terraform configuration

Step 7: Configure AWS credentials for GitHub Actions

Step 8: Create the GitHub Actions workflow

Step 9: Deploy the Go application to AWS Lambda

Step 10: Test the API

Conclusion

How does `traceroute` work?

Visualizing `traceroute`