DEV Community: Mya

Extending my Security System with a Raspberry Pi Network Bridge

Mya — Wed, 16 Jun 2021 15:37:55 +0000

It doesn't happen a lot, but every so often I come across a device that isn't wi-fi supported. This latest case was my security system. On one hand, I like that my cameras aren't taking up bandwidth on my home network and that the system is largely a closed loop. On the other, not having access to my security system without having it tethered into the router is a bit of a pain. For one, my home networking setup isn't that elegant (yet). Second, the last thing I want to do is have more stuff out in the open, co-located with my router. So I decided to get a little creative. Sure, I could've bought a wi-fi adapter, but where's the fun in that. On top of that, I had some other reasons:

I didn't want to spend money on an adapter for this system. Even though they aren't that expensive, I would likely need to wait for one to come in which would probably happen after I left for my trip.
Eventually, I want to do some real-time video processing and having a device closer to the box is promising. This would let me process data straight from the box rather than needing to transmit all that information over wi-fi to another machine.
Finally, I already have more than a dozen pis around. For the first version of this, I wound up using a 3b+. I would likely upgrade this later on when I add the real-time processing.

Today, I show how I set up and configured a Raspberry Pi to act as a WAN client for a connected device. There are a handful of similar guides out there, but finding one for this specific direction / configuration was difficult. Here are some similar guides I used as reference:

Of these guides, "Raspberry Pi 4 Model B WiFi Ethernet Bridge" is the closest. Unlike many of these guides, I run an Ubuntu arm64 image instead of Raspbian OS. The process is largely the same, but some tooling is different. For example, Ubuntu works with netplan by default. In addition to that, we have a slightly higher resource footprint, but it's not the biggest concern for what this little one is doing. To help provide a little context as to what's going on here, I've put together this diagram (and alt-text):

Alt-text for diagram:

At the top, there's a wi-fi router that's connected to the public internet who's wlan0 interface holds the 192.168.4.1 IP address.
Connected to the router over wi-fi are two machines.
- First is Mya's laptop who's wlan0 interface holds the 192.168.4.10 IP address.
- Second is a Raspberry Pi board who's wlan0 interface holds the 192.168.4.30 IP address and who's eth0 interface holds the 192.168.10.1 IP address.
The Security hub connects to the Raspberry Pi's using an ethernet cable.
Some number of cameras connect to the security hub using a cable.

Flash the Image

For simplicity, I used my cloud-init base from my rpi-cloud-init repository to flash my Raspberry Pi (w/ wi-fi access). This gives it a similar look and feel to many of the other microcomputers I have around the house. In addition to that, I get some reasonable default security configurations with that setup (i.e. private key access, no root, custom user, etc). To get a machine up and running, follow Steps 1 - 4 on the projects README.md.

Once the board is running, we'll need to make some additional modifications to the networking setup. The network-config provided in the cloud-init setup generates /etc/netplan/50-cloud-init.yaml. We will be creating a new /etc/netplan/60-static-eth0.yaml file with the following contents.

network:
  version: 2
  ethernets:
    eth0:
      dhcp4: false
      dhcp6: false
      addresses: [192.168.10.1/24]
      gateway4: 192.168.10.1
      nameservers:
        addresses: [192.168.10.1]

This configures the devices eth0 interface to use the static IP 192.168.10.1. With the new file, we need to generate and apply the changes.

$ sudo netplan generate
$ sudo netplan apply

We can verify the changes by inspecting the ifconfig (you may need to apt-get install net-tools).

$ ifconfig
mjpitz@ip-192-168-4-30:~$ ifconfig
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.10.1  netmask 255.255.255.0  broadcast 192.168.10.255    ###### expected change

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0

wlan0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.4.30  netmask 255.255.255.0  broadcast 192.168.4.255

Software

In order to get this to work, I needed to install some basic software components.

isc-dhcp-server is responsible for providing configuration to machines on the underlying network.
dnsmasq intermediates DNS queries between connected devices and the upstream router.
netfilter-persistent and iptables-persistent preserve firewall and iptables rules on reboot (respectively). We will be using iptables to act as our core routing layer.

You can install these components using apt-get.

$ sudo apt-get update -y && sudo apt-get install -y isc-dhcp-server dnsmasq
$ sudo DEBIAN_FRONTEND=noninteractive apt install -y netfilter-persistent iptables-persistent

Once installed, we'll need to configure each system. Note, dnsmasq may be having trouble starting up. This is OK, we'll fix it.

dhcp

First up, let's configure dhcp. dhcp delivers network configuration to devices attempting to connect to a given router. For some reason, I couldn't get the system to work without this installed. I know dnsmasq supports dhcp, but without this process I was getting an error and couldn't obtain a network address.

dnsmasq-dhcp[3685]: DHCP packet received on eth0 which has no address

First, we'll want to modify /etc/default/isc-dhcp-server to point INTERFACESv4="eth0". This will ensure that the dhcp server responds to dhcp requests (similar to how your wi-fi router works). Next we need to configure the dhcp server to know about the subnet that we're allocating to it. To do this, we'll edit /etc/dhcp/dhcpd.conf to contain the following contents:

# communicates that the Raspberry Pi will act as a router for requests.
host router {
  hardware ethernet "mac of eth0, obtained from ifconfig.eth0.ether attribute";
  fixed-address 192.168.10.1;
}

# communicates how to manage the associated subnet(s)
subnet 192.168.10.0 netmask 255.255.255.0 {
  range 192.168.10.2 192.168.10.254;
  option routers 192.168.10.1;
  option dns-name-servers 192.168.10.1;
}

Once dhcp has been configured, we'll need to restart the service.

$ sudo service isc-dhcp-server restart

Once the dhcp server has been restarted, it should be operating with the new configuration. We can verify that it's running properly using sudo service isc-dhcp-server status or by tailing logs with sudo journalctl -u isc-dhcp-server.

dnsmasq

Next we're going to configure dnsmasq. dnsmasq provides discovery logic for requests via a DNS interface. In the default configuration, it conflicts with systemd-resolved (which is why the service likely couldn't start). We'll be modifying its configuration to bind dnsmasq to eth0 allowing it to respond to requests there instead of conflicting on wlan0. To do so, we'll backup the existing configuration and create a new file.

$ sudo mv /etc/dnsmasq.conf /etc/dnsmasq.conf.orig

Next, we'll create a new /etc/dnsmasq.conf with the following content.

interface=eth0                              # what interface to bind to.
listen-address=192.168.10.1                 # what addresses to listen on.
bind-interfaces                             # binds to all interfaces, even if we're only listening on some.
server=192.168.4.1                          # sets the upstream router as a dns server for delegation.
dhcp-range=192.168.10.2,192.168.10.254,12h  # configures the lease time for dhcp requests.

Once we configure dnsmasq, we'll need to restart it for our changes to take effect.

$ sudo service dnsmasq restart

Once restarted, dnsmasq should come up without an issue. We can check its status using sudo service dnsmasq status or by tailing logs with sudo journalctl -u dnsmasq. Before requests can successfully pass through the Raspberry Pi, we need to configure some lower level networking.

Linux networking

The last part of this configuration requires modifying the Linux networking components. To do this, we'll first want to configure the kernel to do packet forwarding for IPv4. Open /etc/sysctl.conf, uncomment the line containing net.ipv4.ip_forward=1, and save. To reload the configuration without a full system reboot, run the following command.

$ sudo sysctl --system

Next, we'll instruct IP tables to allow masquerading over the wlan0 interface. This allows requests to pass through the network interfaces with very little software in between. Once modified, we'll need to persist changes using netfilter-persistent so changes persist between reboots.

$ sudo iptables -t nat -A POSTROUTING -o wlan0 -j MASQUERADE
$ sudo netfilter-persistent save

Verifying client

Once all of this is done, traffic should flow from the connected device through to other devices on the network or internet. We can verify this by connecting a device to the Raspberry Pi. It should successfully negotiate an IP address from the dhcp server (likely 192.168.10.2). In addition to that, we should be able to make requests from the client (i.e. the security system) to other devices on the network or internet.

Once I connected my system, I saw that it successfully obtain an IP from the server using the UI they provide. After, I sent a test email to make sure the request would successfully go through. While doing this, I brought up a terminal on the Raspberry Pi to watch the traffic flow (sudo tcpdump -X -i eth0). At this point, I'm leagues ahead of where I was before. When I went to set up cloud backups, I learned that needed to be done through their mobile app. As a result, I needed to expose the client application ports through the Raspberry Pi in order to connect from my devices.

Exposing ports

The system I bought exposed two ports of interest to me. 9000 provides a media application that can interface with my security systems mobile application. And 554, which exposes a real-time streaming protocol (RTSP) that allows for remote observation of cameras. To quickly proxy these ports, we can use our good ole friend iptables again. By executing the following commands, we can successfully route requests from a port on the Raspberry Pi to a port on the client. Note, ${CLIENT_IP} should be the IP your client obtained. ${PORT} would be the port you want to expose.

$ sudo iptables -A PREROUTING -t nat -i wlan0 -p tcp --dport ${PORT} -j DNAT --to ${CLIENT_IP}:${PORT}
$ sudo iptables -A FORWARD -p tcp -d ${CLIENT_IP} --dport ${PORT} -j ACCEPT

Note that in my case, the RTSP port needed to also be exposed over udp for better streaming support. If you're exposing ports through a Raspberry Pi, you'll need to verify what protocols to expose them over. Also, don't forget to save your updated IP table rules, otherwise changes will be lost on reboot.

$ sudo netfilter-persistent save

Conclusion

That was it. Once everything was setup, I was able to connect to the real-time stream from my devices and monitor my whole system. The mobile application also connected to the security system fine and was able to let me configure cloud backups. While this is where I stop today, I have some more plans for this in the future. As I prepare for some upcoming trips, I want to make sure I can access this system outside of my home network safely and securely. While I don't expect this to become a regular pattern I'll deploy, it was extremely useful to set up in this case.

If you plan on doing this on your own network, be sure to use the proper values for your router and network topology. For example, my wireless system uses 192.168.4.1. Yours might be something different. When it comes to configuring the network for the Raspberry Pi's ethernet port, you can pick any unused range on your network.

~ Thanks for stopping by and I hope you enjoyed the read!

Exploring Redis High Availability

Mya — Thu, 22 Apr 2021 14:50:02 +0000

Recently, I've found myself using Redis for more of the projects that I work on. Redis can be used in a variety of ways. It provides functionality for queueing, set operations, bitmaps, streams, and so much more. Yet, most of my experience with Redis has been as a best-effort cache. Since it's become a staple in my development, I figured it would be good to brush up on its operations.

In this post, I dive into the variety of ways Redis can be deployed. I'll cover the benefits, tradeoffs, and even some uses for each deployment. Finally, I'll describe the deployment that I recently put together.

What is Redis?

Redis is an open-source, in-memory database licensed under the BSD license. It has built-in replication, common eviction strategies, and even different levels of on-disk persistence. Redis also supports modules, which makes it easy to plug in new functionality and interact with native data types. For example, the RedisGraph module extends Redis by providing graph database capabilities. Due to the variety of data structures and its flexibility, it's no wonder Redis has become as popular as it has.

How can Redis be deployed?

One of the big benefits to Redis is that it can be deployed in a variety of ways. We won't dive into every deployment, but we will focus on the common patterns.

Replicated

The easiest deployment of Redis is a replicated architecture. In this deployment, Redis has a single, primary write point called a "master". Any data written to this primary propagates to replicas asynchronously. The diagram below depicts how an application may interact with this deployment.

While the deployment is rather straightforward, there are a few things that we need to keep in mind. First, applications need to be able to track read-only and read-write connections. They could use a single connection, but that likely means that the primary handles every request. By using a single connection, your application makes no use of the replicas that you have in place. Second, this deployment has a single point of failure, the primary node. Should the primary node fail, applications can no longer write data to Redis.

This deployment works well for offline population. For example, maybe you have a background task that's responsible for populating the cache. Meanwhile, your web service knows about the replicas and actively reads from them. Should the primary crash, your background process can simply wait for it to return.

Using Sentinel

To help alleviate the single primary node in the previous deployment, Redis provides Sentinel. Sentinel is a separate process that determines when a given primary node is no longer available. It does this by using a quorum to vote on member availability. When a member becomes unavailable, Sentinel is able to elect a new primary for the cluster. The diagram below depicts how an application may interact with this deployment.

Similar to the last deployment, this approach has some downsides. When a client wants to write data to a Sentinel cluster, they first need to consult the Sentinel process.
Sentinel reports to callers who the current primary in the cluster is. Then, clients will call the primary to write the data. As a result, clients need to be aware they are talking to a Sentinel cluster. This often requires additional configuration that few projects support.

Another risk to this deployment is a potential split-brain state during upgrades or node failures. A split-brain state happens when two nodes cannot come to an agreement. This case is less likely if you manage Sentinel deployments separately from Redis. If you run them in the same pod or on the same host, the likelihood of this scenario increases. To illustrate let us consider the above example.

Suppose we upgrade our cluster and redis-0 is taken down.
Sentinels detect that the pod is no longer available and start figuring out a new primary.
The sentinel for redis-1 votes that redis-2 should be the new primary.
The sentinel for redis-2 votes that redis-1 should be the new primary.
With a quorum of 2, the instances are unable to come to consensus.
Steps 2 through 5 repeat until the instances reach a quorum.

Unfortunately, this is how Bitnami configures the deployment of Sentinel in their redis Helm chart. While split-brains can occur, they may not be as common as you may think. Still, it's something you need to account for in your deployment.

Redis Cluster

Redis Cluster is popular amongst those using Redis for a large amount of data. It shards data across the instances in the cluster, thus reducing the resource requirements of each individual node. Redis does this by hashing the provided key into a slot. Slots are distributed across the primaries in the cluster, often in chunks. The diagram below illustrates how an application might interact with members of a Redis Cluster.

Similar to the Sentinel deployment, applications need to be aware they are speaking to a Redis Cluster. This is due to how Redis Cluster handles its sharding. Clients are responsible for hashing keys to determine which slot they write to. Then, clients write to the instance responsible for the associated slot. This will allow clients to perform a subset of operations, even in the event of a partial cluster outage.

Clustering with Envoy

An alternative to using Redis Cluster would be to cluster using Envoy. Envoy is a well-known, well-supported, network proxy open-sourced by Lyft. It supports many protocols including HTTP, gRPC, MongoDB, Redis, and more. When using Envoy to cluster instances, envoy handles partitioning the keyspace on behalf of Redis. This means the underlying Redis instances shouldn't have clustering enabled.

In this deployment, Envoy favors availability and partition tolerance over consistency. This makes it ideal for a best-effort cache, but not great for more stateful systems. You might consider setting an LRU eviction policy for Redis to ensure stale, unused data is pruned first. One nice thing about this deployment is that clients can speak to any envoy instance in the cluster. This drastically simplifies the logic on the callers' end when compared to Redis Cluster.

2022-03-21 Edit

I've written up a Redis Helm Chart that makes it easy to deploy a Redis cluster using this pattern. Instead of running Envoy as a sidecar to the Redis server, client applications must embed it into their own application. My helm chart provides a simple function for doing so. For a reference on how to do this, see the Registry Helm Chart as an example.

My deployment

While these approaches can work great, I often found myself wanting something a little different. Ideally, I wanted a deployment that:

Improves the availability of the replicated deployment
Improves the stability of the sentinel deployment
Maintains the simplicity of the replicated deployment
Maintains the consistency of the Redis deployments

Such a deployment would improve task and queue-based systems like Celery in python or Machinery in go. Payloads in these systems tend to be small, relatively short-lived, and/or backed up elsewhere. This means we rarely need to shard data across nodes.

As I contemplated how to do this, I kept my runtime environment in mind. Kubernetes provides many capabilities to engineers. For example, adding leader election to a system is relatively easy to do in Go. The client-go library provides a pre-built leaderelection package. This can automate watching a Lease for changes and periodically attempting to claim leadership. This had given me an idea for a sidecar.

The sidecar would use the Lease API to determine leadership in the cluster.
When an election occurs, the sidecar is responsible for updating the topology of the Redis cluster.
1. Each sidecar acts independently, removing the need for coordination beyond the initial election.
2. Each sidecar takes appropriate steps to handle the change in leadership gracefully.
  - The current leader pauses writes and waits for the newly elected leader to catch up.
  - The newly elected leader waits for syncs to complete before attempting to promote itself to leader.
  - All other followers swap over to replicating from the new leader.
3. Once the topology of the cluster has been updated, the sidecar of the newly elected updates a Kubernetes Endpoint.
  - This Endpoint can be used by clients to guarantee requests get sent to the leader.
  - A separate service can be used to allow reads to any node in the cluster.

Such a system can be seen below.

A nice thing about this deployment is that the mitigation techniques are similar to that of the sentinel deployment. While we aren't using the sentinel to perform the election process, we are doing more or less the same thing. A benefit to this deployment over sentinel is that we remove the call to discover the current leader. Instead, clients get a stable address they can dial and always reach a leader, even during rolling upgrades.

In the few tests I ran, I had seen failover times of 6 to 10 seconds. Definitely still working out some tuning, but all in all, I'm satisfied with how it came out. For now, I'm going to keep the project closed source. If you're interested in chatting through this a bit more feel free to reach out.

Anyway, thanks for checking in. I know it's been a while. I hope you learned something while you were here. Thanks for reading!

Repository impressions and conversions

Mya — Mon, 03 Aug 2020 14:23:27 +0000

As someone who works on open source a lot, I often have questions about the reach a given project has. Understanding user behavior around a given product helps companies improve their product and experience. Why should open source repositories be any different?

In this series, I walk through setting up Google Analytics to track metrics around a repository hosted on GitHub. This approach follows a similar technique that's applied to tracking email open rates.