DEV Community: Odysseas Lamtzidis

How to monitor your Ethereum Node in under 5m

Odysseas Lamtzidis — Mon, 02 Aug 2021 15:29:33 +0000

This piece is a blog post version of a workshop I gave at EthCC about monitoring an Ethereum Node using Netdata.

Disclaimer: Although we use Netdata, this guide is generic. We talk about metrics that can be surfaced by many other tools, such as Prometheus/Grafana or Datadog.

The contents are as follows:

Introduction to Ethereum Nodes
What is Netdata
How to monitor a system that runs go-ethereum (Geth)
How to monitor go-ethereum (Geth)

Ethereum Nodes

Running a node is no small feat, as it requires increasingly more and more resources to store the state of the blockchain and quickly process new transactions.

Nodes are useful for both those who develop on Ethereum (dapp developers) and users.

For users, it's crucial so that they can verify, independently, the state of the chain. Moreover, using their own node, they can both send transactions and read the current state of the blockchain more efficiently. This is important, as a range of activities require the lowest of latencies (e.g MEV).

For developers, it's important to run a Node so that they can easily look through the state of the blockchain.

Given this reality, services like Infura or Alchemy have been created to offer "Ethereum Node-as-a-Service", so that a developer or user can use their Ethereum Node to read the chain or send transactions.

This is not ideal, as users and developers need both the speed of their own node and the lack of dependency on an external actor who can go offline at any time.

Running the Ethereum Node

Thus, running an Ethereum Node is not as a fringe activity as one outsider would expect, but rather a common practice for experienced users and developers. On top of that, running an Ethereum node is one of the core principles of decentralisation. If it becomes very hard or complex, the system becomes increasingly centralised, as fewer and fewer parties will have the capital and expertise required to run a node.

Geth is the most widely-used implementation of the Ethereum Node, written in Go.

The Netdata Agent

The Netdata Agent was released back in 2016 as an open-source project and since then it has gathered over 55K GitHub ✨.

TL;DR of netdata monitoring:

You run a single command to install the agent.
Netdata will auto-configure itself and detect all available data sources. It will also create sane default alarms for them.
It will gather every metric, every second.
It will produce, instantly, stunning charts about those metrics.

In other words, you don't have to setup

a) A dashboard agent
b) A time series database (TSDB)
c) An alert system.

Netdata is all three.

How to monitor your Ethereum Node

EthCC was a blast, not only for the energy of the ecosystem, but also for how our workshop was received by node operators from a dozens of projects.

I was stunned to see how many professionals are struggling with monitoring their infrastructure, often using some outdated Grafana Dashboard or the default monitoring system of a cloud provider.

Let's get right into it.

Preparation

The first order of business is to install netdata on a machine that is already running Geth.

Note: Make sure you run Geth with the --metrics flag. Netdata expects the metric server to live in port 6060 and be accessible by localhost. If you have modifed that, we will need to make a configuration change in the collector so that we point it to your custom port.

To install Netdata, run:

bash <(curl -Ss https://my-netdata.io/kickstart.sh)

Visit the Netdata dashboard at <node_ip>:19999.

For illustration purposes, we run a public test Geth server at http://163.172.166.66:19999.

Action plan

We will not cover every single metric that is surfaced by Netdata. Instead, we will focus on a few important ones.

For these metrics, we will:

Talk about what the particular system metric means in general.
Discuss how to read these system metrics, no matter the workload.
Analyze how Geth affects these system metrics.

How to read the dashboard

The dashboard is organized into 4 main areas:

The top utility bar. Particularly important to access the time picker and running alerts.
The main section where the charts are displayed.
The right menu which organizes our charts into sections and submenus. For example, the system overview section has many different submenus (e.g cpu) and each submenu has different charts.
The left menu which concerns Netdata Cloud.

System Overview section

First, we take a look at the System Overview section.

full-resolution image

Top-level Gauges

It has a nice review of the whole system. During sync, we expect to see elevated Disk Read/Write and Net inbound/outbound. CPU usage will be elevated only if there is high use of Geth's RPC server.

CPU utilization chart

full-resolution image

IOwait dimension
It's the time that the CPU waits for an IO operation to complete. It could be running other things, but it doesn't.

How do I read this?
High iowait means that the system is iowait constrained. Usually, this is related to Hard Disk work, but it could be other hardware as well. If I see a consistently low value, that means that I use the CPU efficiently.

softirq dimension

It's the time spent on hardware interrupts handlers. For example, network code is very softirq heavy, as the CPU spends time in kernel mode to handle the network packets.

How do I read this?
It should be very low. Consistently high values mean that the system is not able to keep up with (probably) the network traffic.

CPU Pressure Stall Information (PSI) chart

full-resolution image

In the abstract, it's a measure of how much time is spent waiting for a resource to become available. The CPU could run other tasks, but can't find an available CPU core.

This is only available on Linux systems. FreeBSD and MacOS don't support this, thus you won't find this chart on these systems.

How do I read this?
If you are not utilizing 100% of your CPU, this should be zero. Keep track of this for a couple of days to see the whole range of the "expected" spikes. You can set a new alert for a spike beyond the highest spike under normal load, that way you will know when an abnormal load is detected.

CPU Load chart

full-resolution image

It's the running average of the processes that are waiting for resource availability. Historically, it has been the only measure of CPU performance issues.

The difference with CPU PSI:

Load measures how many processes are waiting for resource availability, while PSI measures how much time applications are waiting for resource availability.

Generally speaking, we care more about PSI than Load. If we are going to use Load, we should keep track of load1 because by the time the other running averages are high, then it's already too late. The system is already throttled.

A rule of thumb is to set an alarm for the following value: 8(or 16)*number_of_cpu_cores. Note that this can greatly vary (even 4 times could be too high) and it's possible that by the time the alert is raised, that you can't interact with the system due to the load.

How Geth affect the CPU charts

Regarding the CPU utilization chart, I see iowait at about ~17%. It's the first evidence that something is not right in my Geth server. Either the network or disks are throttling my system. I see softirq almost non-existent, so disk becomes even more suspicious.

I see about 1-2% of PSI. It should be zero, as the CPU is at about 30% of utilization, but it's not a bottleneck. Most probably, it means that Geth could be more optimized.

As soon as I start spamming my Geth node with RPC requests, I see a considerable bump in both the CPU utilization gauge and the PSI chart. By stress-testing my node, I can set sensible alerts.

In the following image, we can easily identify the time at which I started the RPC request spam to my Geth node.

full-resolution image

Disk Charts

full-resolution image

Disk IO chart

The first chart measures the DiskIO. It's necessary to run Disk benchmarks to truly find the peak of your system and set the alerts accordingly.

How do I read this?
First I run my benchmarks to understand the peak performance of the disks. If I observe that during normal load the disk consistently reaches near the peak performance, then what I do is probably disk io bound and I need to upgrade my disk.

PageIO chart

It measures the data that is pulled from memory. Usually, it's close to DiskIO.

Disk PSI chart

Conceptually, it's the same as CPU PSI. The amount of time that processes are waiting in order to be able to perform DiskIO.

How do I read this?
The charts should be zero most of the time. If they are consistently non-zero, then the disk is a limiting factor on the system and we need to upgrade it.

Important Note
Viewing your Netdata dashboard is actually heavy in Disk IO, as data is being streamed directly from the system to your browser. That means that you will need to look at this chart at a time when you weren't viewing the dashboard.

How Geth affect the Disk charts

This is the most clear indication that something is off with my disks.

The Disk PSI is about 30%, which means that for about 1/3 of the time, some tasks are waiting for Disk resources to be available. That means that my Disks are simply not fast enough.

To verify the correlation with Geth, I can simply stop the process and see the PSI decreasing considerably.

RAM charts

full-resolution image

RAM utilization chart

It's the absolute physical memory in use.

How do I read this?
Ideally, I don't want to see anything listed as free. If I have a lot of free memory, that means that I have more memory than I need. used should be approximately a bit above 50% and it shouldn't be a lot larger than cached.

cached is memory that is used by the kernel to cache disk files for faster access. It is not used, as the kernel will use that memory if a process requires it.

If buffers are very high, that means that the system is under heavy network load. Even in a large server, buffered should be a couple of hundred MBs. buffers are used to store network packets to be processed by the CPU.

Note
A system where the main application is taking care of memory caching (instead of the system) could have a lot of used and almost no cached. This is very rare and probably does not concern most of us.

RAM PSI chart

Conceptually, this is the same metric as CPU PSI.

How do I read this?
If RAM PSI is consistently above zero, then the speed of my memory modules is a limiting factor. I need to get faster (not bigger) RAM.

RAM swap usage chart

When the system can't find the memory it needs, it creates files on the hard disk and uses them as a sort of very slow memory.

Note 1:
It's worth noting that mac, Linux, and FreeBSD have an unintuitive use of swap. They will remove the swap files when no running process is referencing them, not when memory is free. That means that a long-running process will continue to use swap files even if there is available memory.

To solve this, we should either reboot the system, restart the processes, or disable and enable swap.

Note 2:
If you don't see the swap chart, that means that the machine has no swap enabled. Netdata will not show charts that have zero values.

How Geth affect the Ram charts

Geth is really gentle on RAM, consuming what we define in as command line argument. Since there is no swap, we can safely assume that we don't need more RAM with the current configuration.

Moreover, since the RAM PSI is about 3%, I can safely assume that my RAM is fast enough for this workload.

Network charts

full-resolution image

Total Bandwidth chart

It's the total actual data that is being sent and received by the system.

How do I read this?
You need a baseline to read this. If you have consistently more traffic than expected, then something is off.

Important Note
Viewing your Netdata dashboard is actually heavy in network usage, as data is being streamed directly from the system to your browser. That means that you will need to look at this chart at a time when you weren't viewing the dashboard.

How Geth affects the Network charts

We should care about these charts only if they go out of the ordinary (e.g DDoS attack). Observe the baseline of the system (e.g mine is about 1-2 megabit/s) and set the alerts for above the highest spike.

Softnet chart

It counts network receive interrupts processed by the kernel.

How do I read this?
We mainly care about 2 dimensions that should be zero most of the time. If you can't see them, that's a good thing, as Netdata will not display dimensions that are 0.

dropped should always be zero, if it is non-zero your system is having serious issues keeping up with network traffic.
squeezed should be zero, or no more than single digit. If it’s in the double digits or higher the system is having trouble keeping up, but not to the point of losing packets.

Personal computers that have been converted to homelab servers usually have non-zero dimensions, as they are not designed to handle a lot of network bandwidth.

How Geth affects the Softnet chart

In reality, it is the other way around. If we see a high number of dropped or squeezed packets, that could explain strange Geth behavior. It simply is not receiving packets that it should!

Disks section

full-resolution image

The disk section is organized into submenus, one for each Disk.

In my case, I use a block-volume SSD, called sda and mounted on /mnt/block-volume.

Disk Operations chart

full-resolution image

The number of completed operations on the disks.

This is important because it's more taxing on the system to read/write the same amount of data in a high number of small operations, rather in a few larger ones.

The disk may be able to keep up with the write/read IO bandwidth, but not with the amount of operations that are being requested to perform that particular IO bandwidth.

IO backlog chart

full-resolution image

The backlog is an indication of the duration of pending disk operations

How to read this?
On an ideal system, this should be zero. In practice, this sill is non-zero every now and then, simply because of the IO that the system has.

It's relevant to the baseline of the system. You want to see observe the graph for a specific period and set your alerts above the peaks that you see.

Note that if you run backups, these are particularly taxing on IO, so you will need to take those peaks into consideration.

How Geth affects the Disks charts

The first order of business is to locate the disk that is used by Geth to store it's data.

We first see an increased utilization. If that utilization is approaching 100%, that is a clear indication that the Disk can't handle the traffic that is being sent by Geth.

This will most likely result in Geth not syncing.

Continuing, we go to IO backlog and we see what we expected. There is about 100ms constant IO backlog. Our Disk simply can't perform fast enough. It's good that the backlog is constant, that means that the Disk can keep up (but not fast enough for Geth to sync). If the backlog was ever increasing, it means that the Disk can't keep up.

Now that we are sure of the bottleneck, we can observe the other charts to understand better why Geth is hammering our Disks. From what I see the Read/Write IO bandwidth is not terribly high.

A closer examination will bring us to the chart disk_ops, which shows the number of operations/s that the Disks performs. It does about 500 operations per second on the test machine, which would explain why the disk can't keep up.

It's not a matter of how much data is read/written on disk, but rather in how many operations that data is read/written. Geth does a lot of small operations that can be taxing on the disk.

Networking Stack Section

full resolution image

tcp chart

It shows TCP connection aborts.

How do I read this?

All the dimensions of this chart should be zero. If there are non-zero dimensions, that means that there is something in the network, that is not behaving well (e.g a router, the network card on the system, etc.) Consistently high numbers point to a bad network card and you will need to change that.

High numbers of connection aborts mean that your system can't handle the number of connections, probably due to low available memory.

High numbers of time-outs mean that there is some error in the network path between your systems and the system with which you are having the connections.

How Geth is affecting the tcp charts

Geth is a highly networked application, with peers connecting and disconnecting all the time. It's expected to have some baddata, but it shouldn't be worrying. If you observe elevated values, and it originates from Geth (e.g it lowers when Geth isn't running), it's good that you open a GitHub issue on the GitHub repository of Geth.

Applications Section

full-resolution

Interestingly, this section has the same group of metrics that are available in the System Overview Section. The difference is that they are grouped in a per application group basis.

The application groups are defined in the apps_groups.conf.

The user can customize it by running the following command. We assume that the netdata configuration lives in /etc/netdata. Depending on the installation method, this can vary.

/etc/netdata/edit-config apps_groups.conf

The reason we group different processes into application groups is that the user cares about the "functionality" of a certain application, more than they care about the implementation details.

We care about the "web server", not if it's nginx or appache.

Moreover, the user could care about the aggregate behaviour all the "databases" that live in the system.

How do I read this?

Again, we use a baseline. We let the system running under normal load to define our baseline metrics. All the readings afterward will be against that baseline. Generally, we start from a general observation about the system (e.g high RAM usage) and then move to the applications section to identify which application is misbehaving.

How Geth affects the Application Section

The application section is a great pleace to see the resource utilization of your Geth Node. Currently it will be shown as go-ethereum, but we will group all ethereum clients as ethereum node.

The more interesting chart is in the RAM submenu, where we can verify that Geth is not consuming more memory than we want.

Moreover, in the case of an incident (e.g a CPU utilization spike), we can see the application charts for the same period to verify if Geth is behind the anomaly behaviour. If it is, we can then go to Geth's logs or use the Javascript console to see what happened around that time.

This way, we can trace back an incident to the root cause.

eBPF charts

Netdata offers a handful of eBPF charts out-of-the-box. With eBPF we can see in a per-application basis how the application is directly interacting with the Operating System (e.g how many syscalls it does to do_fork).

Although they are not particularly useful for Node Operators, they are very useful to Developers. Using Netdata, they can verify for example that their application is not having a memory leak or that it's not forgetting to close a file that it opened.

If you are a developer in a Ethereum client, please do check out the eBPF charts. We would be grateful for you to try them in your workflow and share any feedback you may have over Discord or our Community Forums.

Although I won't go into the metrics themselves, here are some resources about eBPF as a technology:

Geth section

As already mentioned, I have created a proof of concept integration between Geth and Netdata. It's a collector that automatically detects a running Geth instance, it gathers metrics and it creates charts for them.

The Geth collector uses the Prometheus endpoint of the Geth node, available at node:6060/debug/metrics/prometheus. To activate the endpoint, we must start Geth with the CLI arguments --metrics.

./geth --metrics.addr 0.0.0.0 --metrics

Chaindata session total read/write chart

full-resolution image

Total data that has been written/read during the session (since Geth's last restart).

Charts for both LevelDB and AncientDB.

Chaindata rate chart

Rate of data that are being written/read. Charts for both LevelDB and AncientDB

Chaindata size chart

The size of the Ancient and LevelDB databases. Useful to gauge how much storage you need.

Chainhead chart

full-resolution

It shows the number of the block of the header and block. header is the latest block that your node is aware of. block is the latest block that has been processed and added to the local blockchain.

If these two dimensions are not the same, then the node is not synced.

Tip

A good addition to this chart is the header dimension from another node (or perhaps some service). Having the view of another node in the network can help us understand if our node is seeing what the majority of nodes are seeing.

P2P bandwidth & peers charts

full-resolution image

The number of peers that your node has and the bandwidth between your node and it's peers.

In general, an optimum number of peers is around 30. This can be set as a command line argument when running Geth.

Reorgs charts

full-resolution image

With all the recent talk about reorgs, these charts will show the number of reorgs that have been executed in our node, as also the number of blocks that were added and dropped.

TX pool charts

full-resolution image

Metrics about the tx pool of our Geth node are not particularly actionable, but rather informational about the kind of transactions that are happening.

Goroutines chart

full-resolution image

The number of goroutines is particularly important. With ~50 peers, you should expect about 500 goroutines, while with ~100 you can expect around 1,500. If you have considerably more, there is some bug in the Geth software and you should raise an issue on GitHub.

RPC chart

full-resolution image

For now, it simply shows how many succesful/failed rpc calls are performed in our node per-second.

A sudden increase in rpc calls can indicate a malicious activity (e.g DDoS). Note that a high number of RPC calls can strain the system considerably and a sudden increase in CPU utilization and CPU PSI will be immediately shown.

Default Alerts

When monitoring a system, it's crucial that you setup alerts. The best monitoring system is the one where you never have to open the dashboard, except for a warning alert of an impeding incident.

The good news is that Netdata comes with a slew of default alerts, so most probably you will not have to set anything up.

To get a sense of the default alerts, visit the test server I mentioned above and click on the alert button.

full-resolution image

How Geth affects the default alert

Geth affects the default alerts in 2 ways:

There is a default alert for Geth, that checks if Geth is synced or not, by simply comparing chainhead_block with chainhead_header. This alert will be raised until the Geth node is synced.
Geth may impose abnormal load on the disk. If Geth is functioning normally and a Netdata alert is raised, that means that we need to change the alert to a new default. Not all workloads are the same, and Netdata uses sane defaults that might not be suitable for some workloads.

How to change a default alert

My test server is constantly triggering the disk_space alert. Let's assume that we want to change that.

full-resolution image

From the raise alert, we can see the source field. From that field, I get three pieces of information:

Netdata's configuration lives in /etc/netdata (it can live in other places, depending on the installation method).
The configuration file that I care about is health.d/disks.conf.
The alert starts at line 12 of the source file.

To change the alert:

ssh into the machine: ssh root@163.172.166.66.
run sudo /etc/netdata/edit-config health.d/disks.conf.
Find line 12.

template: disk_space_usage
       on: disk.space
    class: Utilization
     type: System
component: Disk
       os: linux freebsd
    hosts: *
 families: !/dev !/dev/* !/run !/run/* *
     calc: $used * 100 / ($avail + $used)
    units: %
    every: 1m
     warn: $this > (($status >= $WARNING ) ? (80) : (90))
     crit: $this > (($status == $CRITICAL) ? (90) : (98))
    delay: up 1m down 15m multiplier 1.5 max 1h
     info: disk $family space utilization
       to: sysadmin

The above is the current running alert. I can either comment out the entire alert definition by adding # in front of every line, or change the values.

The alert syntax is out of the scope of this blog post, but our documentation should offer everything you need.

Extending the Geth-Netdata integration

It's trivial to extend the integration between Geth and Netdata. Be it with more charts and alerts or for other Ethereum Clients.

If you want to learn how, read on Part 2: Extending the Geth-Netdata integration.

More Netdata goodies

If you have reached thus far, you might be interested in other Netdata collectors, relevant to the operation of a Geth Node:

smartd monitoring with NVME capabilities being implemented by our community as we speak.
Nvidia GPU monitoring
Experimental - automatic Anomaly detection
Metric Correlations

In conclusion

System and performance monitoring is an extremely complex subject, as there is a high number of interdependencies between the system and the workload. A single issue may surface in a dozen different places, from system metrics to logs and user-facing issues.

Geth in particular is a critical piece of infrastructure, and any possible downtime may have serious repercussions to both the operator and the end user.

For this reason, I want to dig more into the matter and publish more content that helps users and node operators in understanding their systems and work proactively.

I need you!

If you are a Node Operator, I would love to talk to you and learn more about the challenges that you are facing. My goal is to install Netdata on large nodes in production and observe the effects that Geth's incidents have on the underlying systems.

By understanding the deeper interdependencies that Geth has with the underlying system, I hope to educate more users and operators in monitoring their systems and avoiding incidents.

You can find me on the Netdata Discord, our netdata community forums and on twitter!

Kudos

I want to give some kudos to my fellow colleagues @ilyam8, @ferroin and, @kkaskavelis for making all this work possible 🚀

How to extend the Geth-Netdata integration

Odysseas Lamtzidis — Mon, 02 Aug 2021 15:28:49 +0000

How to extend the Geth collector

This is the the last of a 2-part blog post series regarding Netdata and Geth. If you missed the first, be sure to check it out here.

Geth is short for Go-Ethereum and is the official implementation of the Ethereum Client in Go. Currently it's one of the most widely used implementations and a core piece of infrastructure for the Ethereum ecosystem.

With this proof of concept I wanted to showcase how easy it really is to gather data from any Prometheus endpoint and visualize them in Netdata. This has the added benefit of leveraging all the other features of Netdata, namely it's per-second data collection, automatic deployment and configuration and superb system monitoring.

The most challenging aspect is to make sense of the metrics and organize them into meaningful charts. In other words, the expertise that is required to understand what each metric means and if it makes sense to surface it for the user.

Note that some metrics would make sense for some users, and other metrics for others. We want to surface all metrics that make sense. When developping an application, you need much lower level metrics (e.g eBPF), than when operating the application.

Let's get down to it.

A note on collectors

First, let's do a very brief intro to what a collector is.

In Netdata, every collector is composed of a plugin and a module. The plugin is an orchestrator process that is responsible for running jobs, each job is an instance of a module.

When we are "creating" a collector, in essence we select a plugin and we develop a module for that plugin.

For Geth, since we are using the Prometheus Endpoint, it's easier to use our Golang Plugin, as it has internal libraries to gather data from Prometheus endpoints.

The following image is useful:

If you want to dive into the Netdata Collector framework:

Geth collector structure

So, in essence, the Geth collector is the Geth module of the Go.d.plugin.

As you can see on GitHub, the module is composed of four files:

charts.go: Chart definitions
collect.go: Actual data collection, using the metric variables defined in metrics.go
geth.go: Main structure, mostly boilerplate.
metrics.go: Define metric variables to the corresponding Prometheus values

How to extend the Geth collector with a new metric

It's very simply, really.

Open your Prometheus endpoint and find the metrics that you want to visualize with Netdata.

e.g p2p_ingress_eth_65_0x08

Open metrics.go and define a new variable

e.g const p2pIngressEth650x08 = "p2p_ingress_eth_65_0x08"

Open collect.go and create a new function, identical to the one that already exist. Although it doesn't really makes a difference in our case, we strive to organize the metrics into sensible functions (e.g gather all p2pEth65 metrics in one function). This is the function that we will do any computation on the raw value that we gather.

Note that Netdata will automatically take care of units such as bytes and will show the most human readable unit in the dashboard (e.g MB, GB, etc.)

e.g

func (v *Geth) collectP2pEth65(mx map[string]float64, pms prometheus.Metrics) {
    pms = pms.FindByNames(
        p2pIngressEth650x08
    )
    v.collectEth(mx, pms)
    mx[p2pIngressEth650x08] = mx[p2pIngressEth650x08] + 1234

}

func (v *Geth) collectEth(mx map[string]float64, pms prometheus.Metrics) {
    for _, pm := range pms {
        mx[pm.Name()] += pm.Value
    }

We also need to add the function in the central function that is called by the module at the defined interval.

func (g *Geth) collectGeth(pms prometheus.Metrics) map[string]float64 {
    mx := make(map[string]float64)
    g.collectChainData(mx, pms)
    g.collectP2P(mx, pms)
    g.collectTxPool(mx, pms)
    g.collectRpc(mx, pms)
    g.collectP2pEth65(mx, pms)
    return mx
}

Lastly, now that we have the value inside the module, we need to create the chart for that value. We do that in charts.go:

chartReorgs = Chart{
        ID:    "reorgs_executed",
        Title: "Executed Reorgs",
        Units: "reorgs",
        Fam:   "reorgs",
        Ctx:   "geth.reorgs",
        Dims: Dims{
            {ID: reorgsExecuted, Name: "executed"},
        },
    }
    chartReorgsBlocks = Chart{
        ID:    "reorgs_blocks",
        Title: "Blocks Added/Removed from Reorg",
        Units: "blocks",
        Fam:   "reorgs",
        Ctx:   "geth.reorgs_blocks",
                Type:  Line, 
        Dims: Dims{
            {ID: reorgsAdd, Name: "added", Algorithm: "absolute"},
            {ID: reorgsDropped, Name: "dropped"},
        },
    }

Let's explain the fields of the structure:

ID: The unique identification for the chart.
Title: A human readable title for the front-end.
Units: The units for the dimension. Notice that Netdata can automatically scale certain units, so that the raw collector value stays in bytes but the user sees Megabytes on the dashboard. You can find a list of supported "automatically scaled" units on this file.
Fam: The submenu title, used to group multiple charts together.
Ctx: The identifier for the particular chart, kinda like id. Use the convention <collector_name>.<chart_id>.
Type: Line (Default) or Area or Stacked. Area is best used with dimensions that signify "bandwidth". Stacked when it make sense to visually observe the sum of dimensions. (e.g thesystem.ram chart is stacked).
Dims:
- ID: The variable name for that dimension.
- Name: human readable name for the dimension.
- Algorithm:
  - absolute: Default (if omitted) is absolute. Netdata will show the value that it gets from the collector.
  - incremental: Netdata will show the per-second rate of the value. It will automatically take the delta between two data collections, find the per-second value and show it.
  - percentage: Netdata will show the percentage of the dimension in relation to the sum of all the dimensions of the chart. If four dimensions have value = 1, it will show 25%.
  - Mul: Multiply value by some integer.
  - Div: Divide value by some integer.

A final note on extending Geth

The prometheus endpoint is not the only way to monitor Geth, but it's the simplest.

If you feel adventurous, you can try to implement a collector that also uses Geth's RPC endpoint to pull data (e.g show charts about specific contracts in real time) or even Geth's logs.

To use Geth's RPC endpoint with Golang, take a look at Geth's documentation.

To monitor Geth's logs, you can use our weblog collector as a template. It monitors Apache and NGINX servers by parsing their logs.

Add alerts to Geth charts

Now that we have defined the new charts, we may want to define alerts for them. The full alert syntax is out-of-scope for this tutorial, but it shouldn't be difficult once you get the hang of it.

For example, here is a simple alarm that tells me if Geth is synced or not, based on whether header and block values are the same:

  1 #chainhead_header is expected momenterarily to be ahead. If its considerably ahead (e.g more than 5 blocks), then the node is definetely out of sync.
  2  template: geth_chainhead_diff_between_header_block
  3        on: geth.chainhead
  4     class: Workload
  5      type: ethereum_node
  6 component: geth
  7     every: 10s
  8      calc: $chain_head_block -  $chain_head_header
  9     units: blocks
 10      warn: $this != 0
 11      crit: $this > 5
 12     delay: up 5s

You can read the above example as follows:
On the charts that have the context geth.chainhead (thus all the Geth nodes that we may monitor with a single Netdata Agent), every 10s, caluclate the difference between the dimensions chain_head_block and chain_head_header. If it's not 0, then raise alert to warn. If it's more than 5, then raise to critical.

Some useful resources to get you up to speed quickly with creating alerts for our Geth node:

Note that if you create an alert and it works for you, a great idea is to make a PR into the main netdata/netdata repository. That way, the alert definition will exist in every netdata installation, and you will help countless other Geth users.

Here are some useful resources to create new alerts:

Extend Geth collector for other clients

The beauty of this solution is that it's trivial to duplicate the collector and gather metrics from all Ethereum clients that support the Prometheus endpoint:

The only difference between a Geth collector and a Nethermind collector is that they might expose different metrics or the same metrics with different "Prometheus metrics names". So, we just need to change the Prometheus metrics names in the metrics.go source file and propagate any change to the other source files as well.

The logic that I described above stays exactly the same.

In conclusion

Extending Geth for more metrics is trivial.

As you may suspect, this guide is applicable for any data source that is exposing it's metrics using the Prometheus format.

The Network State community

Odysseas Lamtzidis — Wed, 30 Jun 2021 11:26:46 +0000

Meet the Network State community

They say that if you want to go fast, you go alone, but if you want to go far, go together.

Over the last months, there have been more and more voices around the ideas of the Network state.

I was first introduced to these brands of ideas while reading book: "The Sovereign Individual". In the new cyber-world, societies are bound to organize again in merchant republics and city-states. The book was written in the late 1990s, thus the use of "cyber".

It argues that our obsession with nation-states will be equally bizarre in the future, as the medieval oaths are today.

The Internet converts everything to a frontier, as it brings together people that are located all over the globe. In a world where people and capital can move freely, the nation-states will have to compete.
Compete in the sense of keeping capital and people from fleeing. Although violence works for people, capital in the form of cryptocurrency will move either way. Voting with your feet, as Balaji S. Srinivasan notes, will be much more effective than voting with your wallet or vote.

With that in mind, the ideas that have been championed by futurist, startupper, and investor Balaji S. Srinivasan are not that bizarre. Actually, they fit right in.

Balaji's thesis is simple. Geographical proximity no longer equates to cultural proximity. Thanks to the Internet, we can communicate and form online communities, as we already have. Using blockchain as a consensus layer to codify the group's rules, we can form a state. A network state, that is.

First online, then on land.

As he underlines, there is a very simple reason to do this.

We want to be able to peacefully start a new country for the same reason we want a bare plot of earth, a blank sheet of paper, an empty text buffer, a fresh startup, or a clean slate. Because we want to build something new without historical constraint.

So, how do we go about it?

As you can imagine, the domain is vast, covering many different domains, from legal to construction. Thankfully, there are a handful of people already working on such ideas. Creator Cabins is a great example of an MVP project, where they explore the actual creation of small facilities.

There are other, larger-scale examples, like culdesac. They are building a whole neighborhood from scratch, right in the US.

Of course, a neighborhood is a far cry from actual sovereign land, but it's a start.

The community

Now, seeing all these disparate discussions online, I thought that a community would be a great place for people to come together and talk about this domain.

This is a by-definition multi-disciplinary domain. The more diverse the community is, the more fruitful the discussions will be.

The long-run is particularly important, especially if we want a real societal change. A geographically distributed group of people, but who share common values and culture.

We want to bring us together.

To interact with the network-state community, you will need Urbit.

If you haven't heard about it, no worries. It's a complete re-definition of modern computing, so let's talk about Urbit for a bit.

Urbit

Urbit is an entirely new paradigm in personal computers. It's a compact system for an individual to run their own permanent personal server on any Unix machine with an internet connection.

Urbit was created as an exit from a world where applications and services and controlled by big companies. It's not only that they monetize your data, but it's also about the very fact that they can deplatform you at any time.

Urbit believes that the only way forward is for us to run the services and applications. They achieve it by creating an entirely new OS and peer-to-peer network that is simple by design and 100% owned by its users.

It's an entirely new stack, an integrated tool for people to communicate and build communities. A tool that the people themselves control, they can trust and extend to their liking.

Urbit OS

Urbit OS is a new, carefully architected software stack: a VM, programming language, and kernel designed to run software for an individual.

It's completely sealed from the system it runs on. In the Urbit world, every person has their own Urbit OS node.

Urbit ID

Urbit ID is an identity and authentication system specifically designed to work with Urbit OS. When you boot or log in to Urbit OS, you use your Urbit ID.

The Urbit ID is a short memorable name (e.g ~ravmel-ropdyl) that it's a username, network address, and cryptocurrency wallet all in one. Since it's registered on the Ethereum blockchain, it's yours forever. It's cryptographic property.

OS 1

In early 2020, Urbit released OS 1. A minimal interface for group communication happens to be the first complete interface. This is the Eden of our community.

Lanscape, as the interface is called, will be the place where we will collaborate and have our discussions. A calm place for people who search for the truth.

Install Urbit

To join the community, you will have to install Urbit.

The easiest way is to follow the Getting Started Guide and use the Port Application:

Alternative -- Using the CLI

Although the CLI is more technically involved, it's an older, more stable client. It's a good alternative in case you face any challenges with the Port.

macOS

mkdir ~/urbit
cd ~/urbit
curl -JLO https://urbit.org/install/mac/latest
tar zxvf ./darwin.tgz --strip=1
~/urbit/urbit

Linux

mkdir ~/urbit
cd ~/urbit
wget --content-disposition https://urbit.org/install/linux64/latest
tar zxvf ./linux64.tgz --strip=1
~/urbit/urbit

Linux users may need to run this command in another terminal window to access your Urbit on port 80:

sudo apt-get install libcap2-bin
sudo setcap 'cap_net_bind_service=+ep' ~/urbit/urbit

After installation, you should see a block of output that beings with the following line:
Urbit: a personal server operating function

Welcome to your last computer.

Identities

A core concept of Urbit is its Identity. Although we won't go into detail here, it's important to remember that IDs have a hierarchy in the Urbit universe.

There are galaxy IDs, where each galaxy is the parent of many star IDs and each star is the parent of many planets IDs. We, the users, are concerned with planets (and comets).

You can read more about Urbit identities in the docs

Planets and comets

Planets are scarce, as this prevents spamming amongst other things. This artificial scarcity makes planets valuable and thus not free.

If you have already acquired a planet, you can follow this guide.

Comets on the other hand are practically unlimited and free. It's a great way to try out the network for free. They have a very long and unwieldy name

~dasres-ragnep-lislyt-ribpyl--mosnyx-bisdem-nidful-marzod

As we read from the urbit documentation:

You can continue using this comet indefinitely. There are currently few differences between using a comet-level identity and a planet-level one. However, some groups will not allow comets entry in order to maintain a certain level of quality, and changes may be made in the future that further devalue comets. They will always, however, be able to access the basic functions of the network.

A comet also comes with a long and fairly unmemorable name whereas a planet has a short name and a "sigil" (avatar) associated with it that makes it more identifiable on the network. You may notice all this within the first few minutes of using Urbit.

Using Landscape

Landscape is the UI of this new computer. To visit landscape, just visit localhost on your computer.

If it doesn't work, try localhost:8080, which is the fallback port in case the default HTTP port 80 is already taken by another app.

Search for the following output in your terminal. It will tell you the port of your Urbit.

eyre: canceling ~[//http-server/0vu.34ksf/2/3]
http: web interface live on http://localhost:8080
http: loopback live on http://localhost:12322
pier (18836): live

Welcome to the clean slate OS

Run Urbit: Port application

Open the application and choose your comet. Click Launch and that's it!

Note: You can find your Access Key by pressing the bottom left button.

Run Urbit: CLI

We are going now to boot our free identity, or comet. It's very simple.

Run ~/urbit/urbit -c mycomet

Caveat: When you run urbit again, you will not use the flag -c. Thus, ~/urbit/urbit mycomet.

It can take a while to load the comet since we need a unique id. Be patient.

Once it finishes loading, you should see the following line at the terminal

~sampel_marzod:dojo>, where dojo is the Urbit command line.

Before proceeding go ahead and type +code in the terminal. Copy the result, we will need this.

Network State group

To join the community, click on "Join Group" and type ~pitrup-nosfyl/network-state

You should land on an empty page with all the channels greyed out. Welcome to our community.

Simply choose a channel that you wish to join and click on "Join Channel"

Before proceeding to discover the discussions, make sure you leave a comment in the intro channel. It's a great place to get to know each other, not on a personal level necessarily, as we live in the pseudo-anonymous age. But, we would like to know why you joined the group if you are engaged in relevant projects and whatever else relevant.

For updates about the community, you should see the Group Feed.

The community is very new, so everything that you see is open for change. If you feel that something is missing or should be altered, please do say so.

The community is only as good as its members are, so if you think you can improve the shared experience, it's up to you to improve it. Sending me a message is a good start.

Welcome to the future of communities.

1729

A small note before I let you go. Although this group will be focused on ideas around sovereign individuals and the network state, there is already a discord group of 1729ers.

1729 is a project started by Balaji. As he describes it:

It's a newsletter for technological progressives. That means people who are into cryptocurrencies, startup cities, mathematics, transhumanism, space travel, reversing aging, and initially-crazy-seeming-but-technologically-feasible ideas. Basically, if you like @balajis' past work, you will enjoy this

For now, we don't expect to create an Urbit group for 1729, but if there are users who prefer urbit for its more private, decentralised experience, we will consider offering the choice. The goal is to bring people together, not split them up.

To join the Discord group: https://discord.gg/WPN3XftmkV

Relevant Resources

Here is a helpful list of resources that might prove useful, as you enter this domain's rabbit hole.

Update #1 on Learning in Public

Odysseas Lamtzidis — Sun, 13 Jun 2021 13:50:53 +0000

Hello everyone,

It's been 2 weeks since I launched my [Learning in Public] Roam Research graph.

As I wrote in the initial blog post, the goal of the graph is to start documenting my learnings in a manner that is usable by others. Moreover, I hope that this graph will result in a multiplayer learning experiment. That means that multiple people will log their learnings on the graph and enrich the various Locations of Knowledge.

After running with this for 2 weeks, here are a couple of learnings

New Locations of Knowledge

You can find all the Locations of Knowledge on the Roam Research Graph

Solidity

Baseline architecture for an oracle
What is a library?
What is the delete keyword
Fallback Functions
What is delegatecall()
How to force ether into a contract
Security implications for
- Interfaces
- transfer(), send()

Solidity has been going great, having finished CryptoZombies and investing considerable time into Ethernaut.

As you will see, the new additions are vastly more advanced concepts than the first iteration of the Location of Knowledge for Solidity

Mechanism Design

I am happy to share with you all the I started a new course for Mechanism Design. Notes from the 2 first lectures have been added in the Graph.

Mechanism Design is hugely relevant to blockchains, since it's about designing systems that achieve a certain goal in the presence of strategic players.

It's a science that compliments psychology, computer science and game theory. As you can see, with blockchains having an intrinsic economic part, being able to create systems that align the interests of different players is challenging.

A very good example of good mechanism design is MakerDAO. The good design stems from the fact that the system has been able to work for all these years, providing both lending services and a stablecoin that is fully transparent. In it's very essence, the stability is due to good mechanism design and not obscure storage of dollar equivalents

DeFi

I have started learning more about DeFi, thus a couple of interesting insights have been already logged into the Graph.

What is MakerDao
What is Uniswap
How AMM work?
What is slippage?

This is a first pass and I will enrich the content as I read more about them.

System

Minor changes have been implemented in the personal pages.

There are 2 new categories, one to log the Open Questions that the person has while taking fleeting notes. Another to keep track of videos that are relevant to a fleeting note and the person wants to view them alter.

I consider to re-evaluate the system on a monthly basis.

Want to engage?

Take a look inside the graph. If it's interesting to you, drop into Discord and let's have a chat!

Discord: https://discord.gg/CqpaY9FAgU

See you in 2 weeks :)

Multiplayer Learning in Public with Roam Research

Odysseas Lamtzidis — Sun, 30 May 2021 10:23:35 +0000

I have started an exciting new project, that will go in parallel with everything that I am doing. In short, it's called Learning in Public and I have written about it in this blog post [[Learning in Public intro post]].

Learning in public serves 3 main functions, which I am refining every day that I use this system.

It forces me to write down and note every single thing that I write. Although it slows things down, I can later document what I learn.
It's a call for like-minded people. Learning is simply much more fun when learning with others. Moreover, I find it difficult to participate in 10 different domain-specific communities. People are multi-faceted, and I want this community to be too.
It's a great repository to help others get up to speed in the domains that I cover.

TL;DR

Visit the Graph on Roam Research
Read the blog post on Roam Research

Roam Research

As I hinted in the title, I opted to go for Roam Research. The choice was a no-brainer, as it's by far the lowest friction that I had with writing down things. It's snappy, yet powerful.

With the power of [[pages]] and ((blocks)), you can do project management and any kind of organization you want. It needs some experimentation and time, but it's possible.

For those who don't know Roam, the TL;DR version is the following.

In Roam research, every bullet that you see is a block. It has a hierarchal structure. While writing you can either reference entire pages or blocks. When you reference a page, in essence you embed a link to that page, while you create a backlink in that page towards the page that you made the reference. The same for blocks.

In essence, you can use [[pages]] as both a page to write into or a tag to anchor the children. For example, if I have the systems that I use inside a [[project system]] page, I can go to that page and get an overview of all the systems that I am using.

For the same reason I use tags as column names for the kanban. I can easily get a view of all the items that are currently in the DOING column. Moreover, using filters, I can do advanced filtering. For example, all the blocks that have a project DOING parent and Learning in Public. Of course, this particular example is not very helpful, since I could simply visit the project page for Learning in Public, but you get the idea.

At this point it's important to note that a #tag and a [[page]] work exactly the same way. The difference is only visual, perhaps to have them implicitly for different uses.

With these primitives in mind, let's see the actual system that I am using.

The structure

The system can be divided into 2 main groups.

The System: How I use the graph in my every day and how I hope that others will do too

The Ontology: How knowledge is organized in this repository.

Let's start with the ontology, which is basically a cool word to signify "structure". We will see what are the building blocks of this repository and why I chose to organize it that way.

Ontology

The repository uses a set of tags and pages to organize knowledge.

Pages

[[project]]

A project page. Although we don't need to narrowly define a project, let's just say that it's a define set of actions to achieve a particular goal. e.g [[Odysseas learns Solidity]] or [[odyslam.com v3]]

[[kanban]]: It's the main way to store items that I need to go through for that particular project.

We have 3 different groups [[project TODO]], [[project DOING]], [[project DONE]]

Items are divided into IMP:<item_name> and RE:<item_name>

IMP: implement something. We have already decided to do this.
RE: Research into something. We don't know yet if we will implement it or not.

Every item can have a START DATE and END DATE. This is helpful for us to keep track of when we start/finish our project items

[[project system]]: We define the system that supports this project. Systems are much more effective than goals because they define habitual behavior. Read more on [[system vs goal]] and this blog post by [[james clear]]
[[project brainstorming]]: A general category to write thoughts and ideas for a particular project. After we refine and them, we should transfer them into the #kanban as project items.
[[project NEXT ITERATION]]: A general category for things that we want to do, but not at the current [[project version]]. We write them here so that we keep track of them and not forget them.
[[project persona]]: When doing a project, it is helpful to remember for whom this project is for. We assume that the project has a consumer, a human at the other end who will use the outcome of the project (e.g a tool). We should always have an end-user at mind and this category is the perfect place to take a few notes about it.
[[project inbox]]: A query that returns all the fleeting notes about a particular project. This enables the user to keep of a log of all their learning in the same top-level block (e.g a date) and be certain that every fleeting note will end up in the appropriate project.
[[project open questions]]: The same functionality as [[project inbox]], but for open questions.

[[Location of Knowledge]]

page name: "Learn ", where X can be anything
Type: Is it a [[tooling]] or an entire [[domain]]. This is rather ill-defined and we will reassess as more LoCs are created.
Domain: Can we group this into a higher order [[domain]] ? e.g [[Learn Solidity]] is in the domain of [[Learn Ethereum]]
Introduction
How to read this category: An introduction on how to read a Location of Knowledge.
[[insights]]: This is the primary category. A set of curated insights about this category. Usually, they are complementary to some original [[external resources]]. We don't need to mirror the entire original context, only our "insight" or note that we believe is important. This way, we super-charge our learning by coupling original content and "hints" that might be unintuitive or hard to grasp. It's learning 10x.
[[external resources]]: This group is for external resources that are not tied to a particular insight, but are useful to have around.
[[capstone project ideas]]: A capstone project solidifies the understanding of the subject. This is a category to add ideas for capstone projects that people can take up on.

[[personal page]]:

This is the personal space for every user of this repository. It is the source of every page that is created by that user. An anchor for their content.

meta: meta information about the user (e.g twitter account)
fleeting notes: the learning log of that particular user. Only they should be writing in it. It is structured as follows:
- date
- A project page
  - raw learnings, ideally with timestamps (to get a view of how much each took)

example:

[[May 29th, 2021]]
- [[Odysseas learns Solidity]]
  - 22:54 Solidity is just a language for writing smart contracts on [[Ethereum]]. Actually, there are alternatives, such as [[Vyper]]
learning projects: A list of project pages that concern learning. e.g [[Odysseas learns Solidity]]
capstone projects: A list of capstone projects that the user would like to do. This helps the community to get visibility into what every user is working on. Visibility translates into more shared knowledge and enables collaboration.

The use of pages is supported by a very handy #roamcult add-on, called 42SmartBlocks. To install the add-on, you have to copy-paste some Javascript code in your graph, which will run every time you open your Roam Research app. The add-on is very powerful, but for now we use it as a templating engine for our pages.

Instead of copy-pasting the blocks in every new LoC or project page, we run the SmartBlock command which auto populates the blocks. Templates are already available to vanilla Roam Research, but SmartBlock makes the templates dynamic. For example, instead of manually adding the date, we can add the SmartBlock code <%DATE:Today%>. This will be replaced by the page with the date of today. It's like typing /today.

#42SmartBlock project 

Tags: #project

Creation Date: **<%DATE:**Today**%>**

## [[project description]]

### [[project inbox]]

{{[[query]]: {and: [[fleeting notes]] [[<%CURRENTPAGENAME%>]]}}}

### [[project open questions]]

{{[[query]]: {and: [[open question]] [[TODO]] [[<%CURRENTPAGENAME%>]]}}}

### [[project brainstorming]]



### [[project persona]]



### [[project system]]



### [[project NEXT ITERATION]]



#kanban {{[[kanban]]}}

[[project TODO]]

[[project DOING]]

[[project DONE]]


#42SmartBlock loc

## [[Location of Knowledge]] template

Type: 

Creation Date: **<%DATE:**Today**%>**

Domain: 

## Introduction

## How to read this category 

The insights are not meant to be all-inclusive, but complementary resources. Follow the instructions at the start of the Insights.

Insights are meant to **greatly accelerate** your learning process of the original material that they accompany. 

This should be collaborative. If you have any questions, just jump into **Discord:** https://discord.gg/CqpaY9FAgU. 

If you want to enrich the content, let's chat in the **Discord** channel.

If you are unfamiliar with Roam Research, visit [[Learn Roam Research]] to learn more about it.

Think of this as a trunk of a tree. A body of knowledge that branches into different subjects. All subjects (branches) have [[external resources]], so you can research further into the subject.

For example, click on the following image for an example from [[Learn Solidity]], where the insights were generated while following [Crypto Zombies](https://cryptozombies.io/) 

![image](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/zb53e16m98ppptss9772.png)



## Insights

## [[external resources]]

## [[capstone projects]] ideas

## People who [[<%CURRENTPAGENAME%>]]

{{[[query]]: {and: [[person]] [[<%CURRENTPAGENAME%>]]}}}

System

Creating an ontology is cool. But you know what's cooler?

Having a system that can support such an ontology, for multiple concurrent users and a considerable amount of time.

The system is the result of many hours of "doing" Learning in Public and thinking about the best way to keep track of my learnings. It's not perfect, but I feel it's refined enough to consider it an alpha version. I do not have a set of explicitly requirements, but there are some principles that I kept in mind when designing this system:

It should have the minimum possible friction. The user should focus on the note taking itself, rather than the system. The system's success, in a way, is measured by it's transparency. The more transparent it is, the more natural it feels to the user. This is important, as reducing friction is the best way to help yourself develop a habit.
It should support multiple concurrent users. We need a system where multiple people will use it in parallel and it will not result into a chaotic graph.
It should be forward facing. We probably use more tags that we currently needs, but that's ok. I am sure that in the future, new use-cases will emerge and the advanced filtering options will be really helpful. It is impossible to foresee all the possibilities, since we are still at the start of the project.

With the in mind, this is how this system should work:

[[Locations of Knowledge]]

They are the treasure of this graph. Only curated insights and refined blocks should end up. They should be treated as a book, rather than a notebook.

They should follow a specific name pattern: , where X is something to be learned.

[[project]]

When you are in doubt, it's a project. A capstone project is a project. Your personal learning page is a project.

A project can have any name, except if it about learning a particular Location of Knowledge. In that case, use

How to keep notes

The core of this graph is the habit of taking notes as we learn something. The following diagram sums up the process

The process described above, results to the following Personal Page, with one personal learning project named "Odysseas Learns Solidity"

How to add insights in a Location of Knowledge

Which results to the following Location of Knowledge

What about tags and pages

At this point, I haven't define a specific "rule" around when to define something as a page and when not. It is ad-hoc, based on what I expect to want to refer to in the future. For example, it make sense to group together all the various notes and information that I gather around the Ethereum Virtual Machine. It's a good rule of thumb to be cautious, so that the graph is not riddles with meaningless tags that make it visually impossible to read.

Unlinked References

Roam Research has a great feature where it will show unlinked references in a page. That means that if I later create a page "EVM", it will tell me about all the occurrences of the word "EVM" in my graph. It assumes that they refer to this page, even though they don't link to it. This is very helpful to be able to backfill links when you add a tag later in the "life" of the graph.

Future

The graph is still very early in its conception, with less than a month of active usage, so I expect this system to be constantly evolved. The more we use the system and structure, the more we will find optimizations to make. In the future, I hope to have a community of people that all learn in parallel on the same graph.

I believe that people are multi-faceted, thus we shouldn't necessarily cluster in domain-specific communities. The community of Symposium will be a rich community, where people learn about many different things, in a way that the learnings of one can benefit everyone. A community where people love learning and helping one another like good comrades. What we share is our love of learning.

CTA

If you find this interesting, make sure to check out the graph on Roam Research and join the Discord community. If you want to participate send a message and we will talk through it. At the moment, Roam Research demands a high degree of trust, so we will need to work things out first.

I regularly tweet about technology, communities, learning and decentralisation, so make sure you follow me for more.

Learning Solidity, in Public

Odysseas Lamtzidis — Sun, 23 May 2021 16:22:11 +0000

A couple of weeks ago, I finished reading The Sovereign Individual. Written in the late 90s, it talks about the ground-breaking changes that are coming to our society. Things like the Internet and computer technology will radically change our societies.

What struck me most profoundly, was the fact that the authors predicted blockchains. Of course, I am not talking about the technology, but rather the principle. They predicted the coming of permissionless, decentralized exchange of information and value.

After finishing the book, I had quite a few things to reflect upon. The most important question was about my place in a world where winner-takes-all effects are becoming the norm. I concluded, amongst other things, that I had to give the whole blockchain domain a second chance. The first chance was back in 2017-2018, when I invested time researching not-so-great blockchain projects.

Being familiar with the basic principles of blockchain, I opted to start with a practical aspect of the space. After consuming a couple of books and podcasts, Solidity seemed the obvious start. It is one of the first languages for running on a general application blockchain. That blockchain is Ethereum. In Solidity, you create programs that run on the Ethereum blockchain itself. That means that these __smart contracts, __as we called them, run on ethereum full node.

The language is Turing-complete, so we can write programs about a great number of use-cases. From running decentralized exchanges (dex), to defining the rules of an organization (DAO). Once we launch these programs on the blockchain, it is impossible for others to censor or stop them. These "decentralized applications" run on **every **Ethereum miner node.

Let's take a moment to reflect on this unique characteristic.

We can define a system, where:

the inner workings of the system are completely transparent
everyone abides by a pre-defined set of rules
all the rules are visible to everyone and known from the start

Moreover, the smart-contract code is open-source by default. This is because nobody will take part in a system that hasn't shared the source code. Once the source code is shared, it is trivial to verify it.

This means that we have a groundbreaking way to create software and design systems. A way that is open source and fair, with limited asymmetry of information (no hidden rules).

Naval Ravikant, also known as the philosopher investor, has said about Ethereum:

So if Bitcoin is a shared ledger, then Ethereum is a shared computer for the entire world to run its most important applications. Source

Perspective

With that in mind, one can see why people are falling in love with Ethereum and Solidity. It is true though, that there are limitations (e.g the Network can support up to 20 transactions/second). Currently, there are many interesting projects, such as DeFi (Decentralized Finance). DeFi is one of the few use-cases that actually make sense to use a blockchain. Always remember that a blockchain is a very expensive way for counter-parties to agree on something.

Learning Solidity in Public

While learning Solidity, I decided to start another project, called [[Learning in Public intro post]]. In that project, I share my learnings about different subjects as I learn them.

Thus, I decided to share my learnings of Solidity.

Think about it. Why spend hours on the internet chasing down the same answers as I did?

Follow the guide that I have compiled and the insights will cover most of the questions that you may have. The goal is to greatly accelerate your learning process. Thus, it is not about creating original content, but rather curating the already available content.

I will simply lay the material for you, as I did for myself.

Outline

Insights from Cryptozombies Solidity path lesson 1-5

_variable means that the variable is a function's argument. It's a convention to make the code more readable.
Storing a string in Memory vs Storage
When to use memory keyword?
uint variable types:
Ethereum gas cost considerations.
When defining a Solidity Struct you don't have to remember the order. There are different ways to call a specific attribute.
What are events?
String comparison
Function Modifiers
Internal Transactions
Transactions vs Calls
Function Visibility internal, external, public and private
Interfaces
calldata variable storage
What is abi.encodepacked?
Assert vs Require
What is contract inheritance?
Import contract
Security Considerations

📍 Sounds simple enough? Then, Enter the rabbit hole.

Learning in Public

Odysseas Lamtzidis — Sun, 23 May 2021 16:17:31 +0000

TL;DR

Just visit the Symposium graph.

Learn Solidity, in Public

Building in Public

The building in public movement is something that I only recently discovered. You pick a medium, like Twitter, and you treat it as your personal "building" journal. You log as you build, thus you build in public.

There a lot of very successful startups and founders who have perfected this art. For example, @shl from @gumroad shares his insights from running a startup and being a creator. Not only I get to share the thrill of doing something with a sense of mission, but I also learn about startups.

In a lot of ways, building in public shares a lot of attributes with open source projects. You get to build in a way where everyone can see your victories and defeats.

The first upside of all this is that you gather a following which is rooting for your success. Because they care, they will provide feedback, creating a virtuous feedback loop. We can largely group the positive effects of this constant back and forth into 2 categories. Firstly, it is invigorating for the spirit and secondly, it's a source of inspiration. Lastly, I am certain that it's a necessity for effective rapid iteration. It's rocket fuel 🚀 for learning.

With that in mind, today I am announcing an experiment of mine. A system to Learn in Public.

Learning in Public

It's an effort to bring all my learning efforts out in the open. Make them available for everyone to leverage.

I spend a considerable time of my every day learning new things. Instead of learning into a vacuum, I will be logging my thoughts, insights, and "Aha!" moments as I research a particular subject.

Every subject will have a corpus of knowledge, like a **tree trunk **which is the foundation of many insights. Think of them as branches. Each insight clarifies some part of the trunk and offers links for further reading.

That way, you should be able to speed up your learning on the particular subject. This effort should accumulate into a rich repository of interlinked knowledge. I hope that people will enrich it, creating a community in the process.

Learning is fun, especially when you share it with other people. This mental tango is so much more useful and productive than banging your head against the wall. Feedback is a crucial ingredient of learning.

The idea is that it will beneficial to everyone. For starters, it will force me to approach learning in an even more systematic manner. Moreover, it will benefit others by laying out a path for learning a particular subject.

Why learning?

I have always been good at learning new ideas, new concepts. It's something that I enjoy and I suspect that is the very reason that I am good at it.

Everyone knows a person that always seems to know a Wikipedia-style fact about a subject. Well, I am that person. I literally read Wikipedia about everything that poked my interest. I remember vividly that once Russia was mentioned in a discussion and I was baffled because I didn't know anything about the country. I thought to myself: "Awesome, let's spend the next hour learning some facts about Russia".

How is it organized?
What is geography?
What about the economy?

You see, this love of learning about things was not visible to me. It was a natural part of my psyche, distinct but invisible. Like a pane of glass, it exists but you don't "see" it, you see through it. It served me well, as I was able to excel in school and later in the university, without studying too much. Given the number of irrelevant projects that I did, I was gliding through courses.

But, there is no such thing as a free lunch. This "aptitude" to learn things was only available when I wanted to learn something. My discipline ~~was~~ is terrible. It was particularly terrible when I had to do something that I considered unimportant.

For that reason, I was never great. I was good if and whenever I decided to be. I assume this is because I never observed my inclination in learning. This ability to learn and place knowledge in a mental map, retrieving it at will.

For me, a big chunk of learning how to learn has been the ability to quickly find the right content. Being able to leverage the power of the internet to identify and consume the next piece of the puzzle.

A revelation

In a way, this is still the way that I am operating, working day and night for the things that I feel passionate about. Thankfully, life is malleable. I am able to work on what I love, doing what I love, and learning about the things that I love.

I do not need to remember the content of a piece of information. I only need to remember the fact that it exists and the metadata that will help with its retrieval.

With google always being at arm's reach, I can vastly increase the amount of available data that I have. I need only to remember the metadata to retrieve them. In essence, I compress the entirety of the information into a handful of keywords.

The thing is, that I was completely unaware of this system until pointed out by a colleague and friend.

During a discussion, he mentioned that I always seemed to have at hand material to support my thesis. He thought that I had a library or something.

I was puzzled. For me, finding things on the internet has always been so normal and easy, that I thought it was trivial. Well, apparently it wasn't.

I spent a couple of days reflecting on that fact.

Everything fell in place. I decided to start taking detailed notes while researching or learning about things. It was hard because taking notes is far slower than just doing™️, but it was well worth the effort. After a couple of weeks, I had a log of all the things that I had learned. All these insights, nicely stored, filled with comments and links to resources. It was magic really, as you could see both the main corpus of knowledge, but also dive in a particular insight at will.

Sharing is caring

Having compiled a "learning log", I thought to myself: "that could be useful to people!". Chances are that I am not the only one who had these questions while reading that article or tutorial.

You see, I have a thing for tools for knowledge. Others are into BDSM, I am into Roam Research.

Why not create a living, breathing graph of knowledge, I thought to myself.

Instead of creating static artifacts, I will create an interconnected living repository. This interconnection is particularly important, it is knowledge on steroids.

On top of that, we could even bootstrap a community. A community about learning, spanning many different topics. Humans are multi-dimensional beings. We can read about Solidity and at the same time reflect the writings of José Ortega y Gasset.

Enter the Symposium

We start with Learning Solidity but more will follow.

Over the next weeks I will be transferring over the bulk of my notes, both from various learning domains and books that I have read.

Follow me here or on Twitter to be up to date!

I opted to name this public space after the Symposium, the famous work of Plato. As we read in Wikipedia,

It depicts a friendly contest of extemporaneous speeches given by a group of notable men attending a banquet

Well, you can think this space as our banquet, where we chat and share our learnings about the things that we are interested in.

I feel that I have written enough. Just visit the Symposium graph and
I will see you there ✌️

Migrating from Nodebb to Discourse

Odysseas Lamtzidis — Thu, 22 Apr 2021 14:07:07 +0000

In this blog post, I note some interesting things that I discovered during the migration of the Netdata Community from Nodebb to Discourse.

I had to deviate a bit from the "official" instructions, thus I thought it would be interesting to share my notes.

Some background

When I joined Netdata in early August, we had just released our forum, based on Nodebb.

According to Nodebb creators:

NodeBB is a next-generation discussion platform that utilizes web sockets for instant interactions and real-time notifications. NodeBB forums have many modern features out of the box such as social network integration and streaming discussions.

The premise of a well-engineered product was proven true, as Nodebb is an extremely pleasant software to work with, offering a modern technological stack which is easily extensible and using npm to vendor it's plugins.

On top of that, the plugins can be installed without rebuilding the forum, meaning that you can swap them on-the-fly. Great!

The reason we chose to move out of Nodebb is that although the project is awesome and the community is vivid, it's not as popular as Discourse. This translates to Discourse having a greater number of available plugins and themes. Our community quickly grew with new requirements that would be hard to accommodate with the existing tools at hand.

In other words, in order to bring the forums up to shape, there is more manual work and maintenance required in Nodebb than in Discourse. Which would have been great, since the choice of customization is really wonderful, but I am the only Developer Relations team member at the moment. Thus whatever development I need, I will do it myself, limiting considerably the time I can invest.

Moreover, as I have now spent considerable time in Discourse, while I do prefer some elements of NodeBB, such as the technology stack or the plugin system; Discourse as a whole offers much more control and options.** It's simply more mature.**

All in all, there is a reason while the grand majority of forums look identical nowadays, and the reason is that Discourse is hard to beat. (Although there are a couple of interesting options oriented more for SaaS products, maybe in another blog post)

The migration

In order to migrate, the good folks at Discourse, with the help of the community, have released a migration tool which parses the MongoDB/Redis database of Nodebb and extracts what can be extracted.

There are a few gotchas which I learned the hard way. Let's see them:

You have to build Discourse from source. As it's a great piece of monolithic software on Ruby, it's not as trivial as one would have hoped.
1. MacOS is not playing nice with Ruby, avoid to reduce unneeded complexity.
2. Go for Ubuntu (or an Ubuntu VM) and prepare to spend quite some time in order to setup everything.
In case of a hosted NodeBB, make sure that you request the entirety of the dump from their support. I spent quite some time trying to figure out why the migration did not work and the reason was that I simply requested half the dump.
In case you encounter the don't know what to do with the file, skipping error, it means that you are using a newer version of mongorestore which has a slightly different syntax than the one that is shown in the migration instructions.
1. Try mongorestore -d <database_name> /directory
2. This will restore the database in a newly created database called <database_name>.
3. Make sure you change the nodebb.rb line concerning the connection to the MongoDB to include the database you defined above: @client = adapter.new('mongodb://127.0.0.1:27017/<database_name>'')
It is possible that NodeBB has created some orphan posts or another bad state inside the database. These abnormalities can break the migration script, bringing the process to a halt. This is what happened to my case and it's the crux of this blog post.

The culprit is 3.14

I used to get some bad states in the forum, meaning that some topics would be created they would be impossible to be retrieved after their creation, resulting in a 404 instead.

As there were already internal discussions about the migration from NodeBB to another platform, I did not spend time to pin down the issue, as it was very very intermittent and I had myriad other things to look after.

After some time, I discovered that the reason for this was the keyword Raspberry pi in the topic name. Every time a topic name included that keyword, the whole topic would enter a bad state, with 404 every single time and manual deletion of the topic (or change of the topic name) as the only solution.

Again, I did not elect to debug the issue any further. That was a mistake.

We found it!

After talking with the good folks at NodeBB for this migration, I informed them of the bug, in case they wanted to dig deeper to evaluate if this is a wider problem or specific to my instance. They were very happy to investigate, and indeed they found the culprit.

They use a specific regex on their reverse proxy that checks for common file extensions that they don't serve. One of these files is .asp which without the ^ or $ qualifier can match with raspberry pi. By updating their rules, they were able to fix this.

But, even with finding the culprit, the bad states had been created inside the database, and unknowingly to me, they would come back and haunt me during the migration.

The migration

When I tried to test the migration, the script was unable to perform the migration and would crash, consistently and without giving much information. With the help of some debugging "print" statements that I put in the script, I found that the script would crash in area of topics import.

At first, it fetches a list of topics, then it starts pulling each topic from the database, and for each topic it pulls all it's posts.

It's simple really.

But, to my disappointment, the script would crash when it it tried to pull the post of a particular topic, the one with the raspberry pi keyword in it's title.

When I tried to fetch the topic from the database, using MongoCLI, I was successful, but when I tried to fetch the first post of that particular topic, I couldn't.

Bingo!

Apparently, for the Raspberry pi topics, while the topic existed, the post did not, resulting in a null return for post_id and the crashing of the whole script. In Nodebb, the post_id of the first post of a topic, is the mainPID.

Here is the data structure of the bad topic:

{
    "_id": ObjectId("5f621635d53e46aab957f13b"),
    "_key": "topic:99",
    "cid": 3,
    "lastposttime": 1600263733946,
    "mainPid": 0,
    "postcount": 0,
    "slug": "99/monitor-pi-hole-and-a-raspberry-pi-with-netdata",
    "tid": 99,
    "timestamp": 1600263733946,
    "title": "Monitor Pi-hole (and a Raspberry Pi) with Netdata",
    "uid": 3,
    "viewcount": 2,
    "thumb": "",
    "deleted": 1,
    "deletedTimestamp": 1600263902138,
    "deleterUid": 37,
    "mergeIntoTid": 100,
    "mergedTimestamp": 1600263902160,
    "mergerUid": 37
}

Now that I knew the exact problem, I had to think of a quick solution.

Here is the function in mongo.rb that fetches the list of topics from MongoDB.

def topics(offset = 0, page_size = 2000)
      topic_keys = mongo.find(_key: 'topics:tid').skip(offset).limit(page_size).pluck(:value)
      topic_keys.map { |topic_key| topic(topic_key) }
    end

I assumed that if I removed the particular topic from that list of available topics, even if it exists in the database, the script will not try to import it and thus the migration will succeed.

I ran: db.objects.find({_key:"topics:tid"}) which returned a large number of documents, like this one: { "_id" : ObjectId("5fb48402d53e46aab9f70b07"), "_key" : "topics:tid", "value" : "198", "score" : 1605665794715 }.

Thus, I needed to delete the document with "value":"99" by running db.objects.remove({_key:"topics:tid", "value": "99"}) and voila, the migration script was able to progress normally.

In case you are interested, you can find the database schema on the NodeBB website.

Here are the steps for the migration in a bite-sized format:

Import mongodump into local instance of Mongo.
Build Discourse from source and setup the discourse instance.
Run script which exports data from mongo, transforms them, and imports them to PostgreSQL.
Export Discourse instance through the Admin panel and import it to the production server.
1. In case you are using a Hosted version of Discourse, you will need to contact support in order to restore the uploaded backup.
Make modifications and prepare a forum for production use.

Final words

The last problem that I encountered during the migration was related to the profile pictures that the script tried to import. Apparently, some were pulled from the Internet and used some special characters which the script could not parse.

As with the Pareto Principle of 80/20, I am pretty happy with an 80% of the perfect result by investing only 20% of total effort. I simply commented-out the profile picture functions and went on with my life, and the migration.

Although this will result in a worse user experience, as existing users will have to re-upload their profile pictures, the other option was not viable as I would have to understand what is the exact issue and change the script, possibly learning some Ruby in the process. I simply don't have the time.

Due to creating the foundations of a Developer Relations program, I have to be ruthless with prioritization, something which I have not done particularly successfully. But, as it's a process, I learn continuously, and thus I opted to go for the quick-and-dirty solution for this.

In the end, what really matters is to improve the experience of the community as a whole, making sure that it grows steadily.

Consider following me on Twitter for more stuff on tech, philosophy and startups ✌️

Introduction to StatsD

Odysseas Lamtzidis — Mon, 15 Feb 2021 14:00:14 +0000

StatsD is an industry-standard technology stack for monitoring applications and instrumenting any piece of software to deliver custom metrics. The StatsD architecture is based on delivering the metrics via UDP packets from any application to a central statsD server. Although the original StatsD server was written in Node.js, there are many implementations today, with Netdata being one of them.

StatsD makes it easier for you to instrument your applications, delivering value around three main pillars: open-source, control, and modularity. That’s a real windfall for full-stack developers who need to code quickly, troubleshoot application issues on the fly, and often don’t have the necessary background knowledge to use complex monitoring platforms.

First and foremost, StatsD is an open-source standard, meaning that vendor lock-in is simply not possible. With most of the monitoring solutions offering a StatsD server, you know that your instrumentation will play nicely with any solution you might want to use in the future.

The second is that you have absolute control over the data you send, since the StatsD server just listens for metrics. You can choose how, when, or why to send data from any application you build, whether it’s in aggregate or as highly cardinal data points. You also don’t need to spend any time configuring the StatsD server, since it will accept any metrics in any form you choose via your instrumentation.

Finally, there is a complete decoupling of each component of the stack. The client doesn’t care about the implementation of the server, and the server is agnostic about the backend. You can mix and match any combination of client, server, and backend that works best for you, or migrate between them as your needs change.

Historically, it has always been easier to measure and collect metrics about systems and networks than applications. In 2011, Erik Kasten developed StatsD while working at Etsy, to collect metrics from instrumented code. The original implementation, in Node.JS, listened on a UDP port for incoming metrics data, extracted it, and periodically sent batches of metrics to Graphite. Since then, countless applications have StatsD already implemented and can be configured to send their metrics to any StatsD server, while the number of available libraries makes it trivial to use the protocol in any language.

How does StatsD work?

The architecture of StatsD is divided into 3 main pieces: client, server, and backend.

The client is what creates and delivers metrics. In most cases, this is a StatsD library, added to your application, that pushes metrics at specific points where you add the relevant code.
The server is a daemon process responsible for listening for metric data as it’s pushed from the client, batching them, and sending them to the backend.
The backend, which is where metrics data is stored for analysis and visualization.

StatsD uses UDP packets because the client/server both reside on the same host, where packet loss is minimal and you can get the maximum throughput with the least amount of overhead. TCP is also an option, in case the client/server implementations reside on different hosts and the deliverability of metrics is a primary concern; in that case, the metrics collection speed will be lower due to the overhead of TCP.

In case you are wondering about the difference between TCP and UDP, this image is most illustrative:

Source

More often than not, an HTTP-based connection is used to send the metrics from the server to the backend, and because the backend is stored for long-term analysis and storage, it often resides in a different host than the server/clients.

StatsD in Netdata

Netdata is a fully featured StatsD server, meaning it collects formatted metrics from any application that you instrumented with your library of choice. Netdata is also its own backend implementation, as it offers instant visualization and long-term storage using the embedded time-series database (TSDB). When you install Netdata, you immediately get a fully functional StatsD implementation running on port 8125.

Since StatsD uses UDP or TCP to send instrumented metrics, either across localhost or between separate nodes, you’re free to deploy your application in whatever way works best for you, and it can still connect to Netdata’s server implementation. As soon as your application exposes metrics and starts sending packets on port 8125, Netdata turns the incoming metrics into charts and visualizes them in a meaningful fashion.

Your applications can be deployed in a variety of ways and still be able to easily surface monitoring data to Netdata. Moreover, Netdata accepts StatsD packets by default, meaning that as soon as your application starts sending data to Netdata, Netdata will create charts and visualize them as accurately as it can. Since there are a myriad of different setups, Netdata offers a robust server implementation that can be configured to organize the metrics in charts that make sense, so you can easily improve the visualization by making some simple modifications.

Because StatsD is a robust, mature technology, developers have built libraries to easily instrument applications in most popular languages.

Python: https://github.com/jsocol/pystatsd
Python Django: https://github.com/WoLpH/django-statsd
Java: https://github.com/tim-group/java-statsd-client
Clojure: https://github.com/pyr/clj-statsd
Nodes/Javascript: https://github.com/sivy/node-statsd

Taking the example from python-statsd, you only need a reachable Netdata Agent (locally or over the internet) and a couple of lines of code. This hello_world example illustrates just how simple it is to send any metric you care about to Netdata and instantly visualize it.

Even with no configuration at all, Netdata automatically creates charts for you. Netdata, being a robust monitoring agent, is also capable of organizing incoming metrics in any way you find most meaningful.

import statsd
c = statsd.StatsClient('localhost', 8125)
c.incr('foo') # Increment the 'foo' counter.
for i in range(100000000):
   c.incr('bar')
   c.incr('foo')
   if i % 3:
       c.decr('bar')
       c.timing('stats.timed', 320) # Record a 320ms 'stats.timed'.

Netdata’s StatsD server is also quite performant, which means you can monitor applications where they run without concerns over bottlenecks or restricting resources:

Netdata StatsD is fast. It can collect more than 1.200.000 metrics per second on modern hardware, more than 200Mbps of sustained statsd traffic, using 1 CPU core

Netdata does this on top of gathering metrics from other data sources. Netdata monitors an application’s full stack, from hardware to operating system to underlying services, organized automatically into meaningful categories. Every available metric is nicely organized automatically into a single dashboard.

Ready to get started?
In the next part of the StatsD series, we are going to illustrate how to configure Netdata to organize the metrics of any application, using K6 as our use-case.

If you can’t wait until then, join our Community Forums where we have kickstarted a discussion around StatsD.

Here are a couple of interesting resources to get you started with StatsD:

StatsD GitHub repository
Scaling StatsD in DoorDash
Netdata StatsD reference documentation

Introduction to community repository: Consul, Ansible, ML

Odysseas Lamtzidis — Mon, 16 Nov 2020 16:46:02 +0000

The post was originally posted on the Netdata blog

On our journey to democratize monitoring, we are proud to have open source at the core of both our products and our company values. What started as a project out of frustration for lack of existing alternatives (see anger-driven development), quickly became one of the most starred open-source projects on all of GitHub.

Fast-forward a couple of years later, and the Netdata Agent, our open-source monitoring agent, is maturing as the best single-node monitoring experience, offering unparalleled efficiency and thousands of metrics, per-second. At the same time, we have gathered a considerable community on our GitHub repository and new forums.

As the community grows, and considering our belief that extensibility is key to adoption, it was only natural to start brainstorming a way to share code and sample applications that supercharge the user experience and the Netdata Agent’s capabilities.

Thus, without further ado, please say hello to our Community Repository.

Although still in its infancy, we expect this repository to be filled by community members who want to share their experience of running Netdata in a production environment or integrated into a technological stack. At the moment, the repository will be used to house all sample applications, which are divided into categories, depending on the use case.

Currently, there are three example applications, all contributed by the Netdata team, which were originally developed for internal use. Let’s take a look at them.

Configuration Management

The first sample application is one I built that focuses on the issue of configuration management of an arbitrary number of Netdata Agents. More specifically, I opted to use Consul, an amazing open-source project by HashiCorp, to dynamically manage the configuration of a Netdata Agent. The keyword is “dynamically”: Whenever I choose to change a configuration variable, the Netdata Agent restarts automatically so that it can pick up the change from the configuration files.

Consul, per their documentation, is a “service mesh solution providing a full-featured control plane with service discovery, configuration, and segmentation functionality”. As such, Consul is routinely used already in cloud-native applications, and it’s ideal for a simple key/value store that we can use to house the configuration variables that we wish to dynamically change. Since Netdata can’t pick up configuration from a RESTful interface, we use consul-template, again an open-source tool by HashiCorp, which watches a Consul node for a specific number of keys, picks up the changes to their values and places them into the templates, generating the changed configuration files in the process.

The code and documentation for this sample application can be found in the specific consul-quickstart directory.

Machine Learning and Netdata Agent’s API

The second contribution came from Andrew Maguire, who contributed a few examples built on the Netdata Agent’s API. The API offers anyone the ability to extract data from the Netdata Agent in an extremely efficient way and build real-time applications on top of it. He leveraged his in-house python library to automatically extract data, add them to panda arrays, and enable live ML, capabilities such as the detection of anomalies.

You can find the examples in the appropriate directory of the community repository and open them in Google Colab. We suggest Google Colab not only because it’s free, but also because they spin up a VM and install all the required dependencies, making it the fastest way to try out the examples and play with the API. To open them on Google Colab, simply open a notebook on GitHub, and click on the Open in Colab button.

Automatic provisioning of Netdata Agents

Last but not least, Joel Hans pulled together the scripts that he had created for him to be able to automatically provision and claim any number of Netdata Agents on remote servers. The sample application is enabled by Ansible, a popular system provisioning, configuration management, and infrastructure-as-code tool. The user defines a set of steps in a .yaml file, called a playbook, and then Ansible is responsible to run this playbook against a number of hosts using SSH as the only requirement.

With Ansible, Joel can install and claim any number of Netdata Agents automatically, so that he can access and monitor his nodes in a matter of minutes, through Netdata Cloud. It’s that easy. You can learn more in the guide.

Now, it’s your turn

The repository is up and running, but we need you to participate. If you are using any of the aforementioned tools and platforms and feel that we could have done something in a better way, please do let us know and make a pull request with your suggestions.

If, on other hand, you are using Netdata with another application that greatly improves the experience, please do create a README about the project and PR it to the appropriate category. The value of this repository is of a compounding nature. The more examples we can get, the more value our users (like you) will be able to receive, and thus the popularity of the repository will invite even more sample applications.

See you all on our repo!

Home webserver setup on a Raspberry pi, using balena and Nginx

Odysseas Lamtzidis — Sun, 26 Apr 2020 16:22:27 +0000

The post was originally posted on my blog, which is actually hosted on this setup, at my home.

Introduction

In this post, we will be using a spare Raspberry pi 4 to host our very own website using the internet connection of our house.

We will start with some introductory terms to get a lay of the land and then we will continue with the tutorial itself.

If you are familiar with the relevant terms (IP, Domain Name, etc.), go ahead and jump to Let's get to it.

Introduction
Table of Contents
- Static Website
- Webserver
- Blogging software - Jekyll
- Domain
- What's an IP
- But what it has to do with domains?
- Dynamic IPs aka "The Plot Thickens"
- Wait a minute, I don't have a domain name
- balena.io
- So what's the deal, exactly?
- Complimentary Software:
- Certbot
- Netdata
- Architecture
- Components
Let's get to it
- Provisioning the Device
- Installing the Software
- Jekyll
- Installing nginx-in-balena
- Configuring the Software
- Environment Variables
- nginx configuration
- ddiclient configuration
- Configuring the environment
- Static IP
- Port Forwarding
- Deploy
- Deploy the project to the device
- Generating the SSL certificate
- Updating your certificates
- Push new content to the website
- Push new content to the website - For advanced users
Comments

Static Website

So, this idea came to me when I was considering alternatives for a new blog I wanted to start. Up to this point, I used Github pages which hosts for free any static website that belonged to an organization, a project or a person.

For those who are not familiar with web programming, as we read from Wikipedia:

A static web page (sometimes called a flat page or a stationary page) is a web page that is delivered to the user's web browser exactly as stored, in contrast to dynamic web pages which are generated by a web application.

So, the website is not supported by a back-end web application, but it's only a set of .html, .css, .js files that the server sends to the browser for the user to view the website. Wordpress sites, for example, are not static, since it's supported by a PHP server, a SQL database and various other components.

Webserver

In our case, to keep things as simple and as lightweight as possible, we will be serving a static website using a static webserver, Nginx. It is one of the oldest and most performant webservers, allowing us to serve up to 1000 users without our Raspberry breaking a sweat.

Nginx is a very robust web server, allowing the user to perform a myriad of different uses, from serving a static website to performing inverse proxy. Its configuration is very straightforward (as we will see below) and as we will find out soon enough and in essence we will simply dictate the server to serve a set of static files (our website), each time there is a connection at a specific port. (What is a computer port?)

Blogging software - Jekyll

As we are lazy, we don't want to write a blog website from scratch, as it would entail considerable overhead for each new post we want to make. What we want, is a framework that will have a certain theme and which will generate the static files of the blog for us, allowing us to focus solely on the content of the blog.

Luckily for us, there is a very easy-to-use framework, called Jekyll. It was created by Github's co-founder Tom Preston-Werner. As we read from the project's repository readme.md:

Jekyll is a simple, blog-aware, static site generator perfect for personal, project, or organization sites. Think of it like a file-based CMS, without all the complexity. Jekyll takes your content, renders Markdown and Liquid templates, and spits out a complete, static website ready to be served by Apache, Nginx or another web server. Jekyll is the engine behind GitHub Pages, which you can use to host sites right from your GitHub repositories.

The power of Jekyll is that it is super easy to use, so easy, that you don't even need programming knowledge (Verified from personal experience). In essence, you configure a Jekyll theme using a central configuration file and then you write the blog posts in markdown format (More on markdown here).

Domain

Now that we have the core pieces of the website, it is time to think about the domain name and the potential issue of dynamic IP.

If this doesn't sound familiar, let's spend a minute for a computer science 101 super-mini-course.

What's an IP

Each computer that is connected to a network is identified by a unique address, or IP, very much like your home address. The Internet is a global network of computers, thus each server has an IP.

As your home router is connected to the internet, it has it's own IP address, which you can find using a service like whatsmyipaddress. At the same time, the router creates a local network in which all your computers at home are connected, thus each computer has a local IP and all your computers, since they are connected to the Internet through the router, will have the same global IP, that of the router.

But what it has to do with domains?

Because it is hard for a person to remember an IP, there are services that have Huge registries, in which an IP is tied to a human-understandable word, or Domain. When you pay for a domain name, you pay to register your IP to these registries and tie that IP to the domain name that you have bought.

Now, each time someone enters that domain name, the computer automatically connects to several Domain Name Registries searching for an IP that is tied to that specific domain name. When it is found, the browser connects to the server using the IP and loads the content.

The problem arises when our IP is not static but it changes continuously, in other words, it's dynamic.

Dynamic IPs aka "The Plot Thickens"

Many ISPs (Internet Service Providers) around the world offer a dynamic IP, meaning that the IP doesn't stay the same but changes now and then, according to the policies of each ISP. This creates a challenge, as the domain name will have to point to a new IP each time our IP changes.

Luckily for us, most domain name providers offer a service called Dynamic DNS. This service allows the customer to use their API to update the IP to which the domain name must point to. We will be using a small program called ddclient which supports most of the known domain name providers.

Wait a minute, I don't have a domain name

If you don't have a domain, go ahead and grab one from one of the major Domain Name providers (just google buy domain name). Make sure that the Domain Name provider that you choose supports Dynamic DNS and ddclient. This guide was tested using Namecheap.

balena.io

At this point, it is apparent that we will be needing to install and configure a bunch of software not only to bootstrap the website but to also keep it up to date. To do that, we will be using balena.io to develop and deploy our software to the Raspberry pi with the ease and speed of using a cloud-service provider.

That's right, using balena to provision and manage our embedded IoT device, we will be having the same tools and workflows that one would expect from AWS.

So what's the deal, exactly?

The team behind balena.io were the first people to port docker to the Raspberry family, showcasing how the container visualization paradigm could serve the domain of the Internet of Things.

Balena now offers a full feature-set that enables us to manage-literally thousands- devices, such as a Raspberry pi, as easily as ever

We will develop our application as a multi-container application, meaning that the distinct services from which the project is constructed will run as distinct containers, completely isolated one from another.

You can think the ~~docker engine~~ balena-engine (our optimized for the IoT version of docker) as an oven where you can bake both a fish and a cake at the same time, while each will taste and smell just fine when you take them out.

In the same sense, each service can run independently, without having to worry with incompatible libraries or different versions. They will ~~taste~~ work just fine.

Finally, balena allows us to

easily access the device's logs
ssh into the host OS or one of the containers
push a new release by simply running a command.

This last part is pure black magic.

You simply define your application in a docker-compose.yaml, you define a couple of Dockerfiles and then you just push your project. balena takes care of building the project specifically for your device on its build-servers and then it simply sends the built project to the device. The smart supervisor is responsible for downloading and setting up your application according to the docker-compose file.

Developing and managing IoT devices have never been so easy and beautiful. Here is a sneak peek of the dashboard for our device:

Disclaimer: I work at balena.io in the product team. Thus, you could say that I am a bit biased.

Complimentary Software:

Certbot

Certbot is a service offered by letsencrypt, a nonprofit Certificate Authority providing TLS certificates for anyone who may ask. This way, users will be able to connect securely on our website, using https.

We will be using the certbot-CLI program to request a certificate for our website. In essence, certbot will place a special file for our webserver to serve. When the authority tests the website, it will find the specific file and verify that the website (and thus the domain) is indeed ours. Giving us a certificate for 90 days.

Netdata

Netdata is a monitoring agent that runs on the device, aggregates, visualizes and presents various data about the operation of the machine. From fairly simple, such as RAM usage, to more complicated such as CPU interrupts. Moreover, it has collectors for specific apps that can auto-detect if it's running and start gathering data.

Although Netdata is fairly complex and customizable, we will be using it because:

It's super light (about 5% CPU consumption) and thus ideal for the constraint nature of a Raspberry pi 4.
It can auto-detect nginx and start gathering data using the stub_page of the webserver. You can read more about stub_page here.

Architecture

We will be using the multi-container functionality of the platform, thus the project will consist of several different containers, each one running a specific component. This architecture enables us to isolate one component from another, facilitating the management and configuration of the application.

Components

webserver: Runs the nginx service and the certbot for SSL generation
ddclient: Runs the ddiclient service
netdata monitoring: Runs an instance of Netdata Monitoring software to overview the load of the server.

Let's get to it

Provisioning the Device

The first order of business would be to provision our device, a Raspberry pi 4.

To do that, we need a new account at balena.io and we need to head over to the Get Started Guide of the balena platform and finish it. It will prompt you to install a demo-app in your device, but that's ok. We need you to get familiar with the platform before we continue.

Disclaimer: After you finish with the guide, don't turn off the device, we will need it for later. Just leave it be. Ok? Cool.

Go ahead, I'll wait.

One Note: The first 10 devices on balena.io are for Free, so using your new account will be more than enough for this project.

Before going forward, we assume that:

You have balena-CLI installed
You have balena-etcher installed
You have logged in balena-CLI
You have logged in balena dashboard

Installing the Software

Jekyll

Although this blog post focuses mainly on setting up a balena-powered raspberry pi webserver, we want to give some insight into how to create a website in the first place.

Visit Jekyll's website and follow the Get started guide.
Get yourself familiar with the Jekyll templating engine
Search for a Jekyll theme that is appealing to you:
1. http://jekyllthemes.org/
2. https://jekyllthemes.io/
Download the theme locally and configure it according to the theme's and Jekyll's documentation.
Build the website according to the theme's documentation, the source files will be placed in a directory called _site.
Upload the files inside _site to a Github Repository.

Disclaimer: If you haven't used Github again:

1) Follow ths guide to create a new repository to Github.

2) Follow this guide to upload all your website's source files to the repository you just created. Simply drag and drop all of them as it is shown in the guide.

3) Congratulations! You have your very first project on Github!

Installing nginx-in-balena

To install the software, we did all the heavy lifting for you. We aggregated all the relevant software and made sure that it can support the Raspberry pi 4 with a balenaOS 64bit. You only have to go download a local copy of the project:


git clone https://repo

cd repo

We are now into the project's directory. Let's configure it!

Possibly, the same software will run without problems on a Raspberry pi 3 with either 32 or 64 bit OS. If you test it successfully on a Raspberry pi 3, please do leave a comment and we will update the blog-post accordingly.

Configuring the Software

Environment Variables

Go here and download the project's repository locally. If you are not familiar with Github, please refer to their documentation.
cd into the repository using a terminal program (cmd in windows).
Open the file .balena/balena.yml using your favorite code editor. If you don't have one, go ahead and install Visual Studio Code
In this file, we need to change a couple of environment variables which we make available to the services via their respective Dockerfiles. Go ahead and change the variables:
1. In order to find the REPO_ZIP_URL, go to your Github repository, click on clone or download and then right-click on Download ZIP and click on Copy Link Address.
2. Fill in the domain of your website according to this certbot example. Please note that www.domain.com and domain.com are two different domains, it is best to include both.

REPO_ZIP_URL=https://github.com/OdysLam/odyslam.github.io/archive/master.zip

REPO_NAME=odyslam.github.io

CERTBOT_MAIL=hi@odyslam.me

CERTBOT_DOMAIN_1=www.example.com

CERTBOT_DOMAIN_2=example.com

This environment variables are not expected to change while the server runs, thus we prefer to define them at build time.

On the other hand, there are 2 environment variables that can be set using balena dashboard and when the nginx container reloads, it will pick them up.

SYNC_WEBSITE: If this environment variable is set to "1", the container will always download the latest version of the website every time it restarts.
CERTBOT_FORCED: If this environment variable is set to "1", the container will always request a new certification every time it restarts. If the current certification is still valid, it will simply inform the user that the certification is up-to-date and will exit.

You can read more about environment variables in balena, in the documentation.
You can read more about build-time secrets in balena, in the documentation.

nginx configuration

Run the commands bellow to generate a private key that will be used by nginx for SSL related functionality. As it might take some minutes, go ahead and read about it in this Stack Overflow Question. Welcome to the world of cryptography.


cd nginx

openssl dhparam -out dhparam.pem 2048

cd ..

Using a text editor, open nginx.conf which will find the nginx directory. Head over to the following excerpt and replace the www.example.com and example.com with your domain name.


server {
listen 443 ssl http2;
server_name www.example.com example.com

server {

listen 80;
listen [::]:80;
server_name www.example.com example.com;

If you want to read more about the nginx configuration file and what the various fields mean, you can read more about it here:

Digital ocean article
Nginx Beginner's Guide

ddiclient configuration

Create a configuration file for ddclient using a text editor:
1. You can find examples in the ddclient's documentation
2. If you used Namecheap, you can find an example configuration in the namecheap documentation
Place the configuration file you just created into the ddclient folder

Configuring the environment

It's time to move on configuring our environment. In our case we need to allow ingoing connections from the router and make sure that the server will always have a static ip in the local network.

Static IP

Balena allows us to set our device to static IP in a breeze.

Find out the IP form that the router uses to assign IPs.
1. Go to your balenaCloud dashboard, you can find the IP of your device at the summary page.
Use a text editor to open the file static-ip that you will find in the directory tools of the repository you downloaded.
Replace the field ROUTER_IP with the IP of the router.
Replace the field DEVICE_IP with the IP of your balena device with a small change. Change the digits after the last . to 100.
Close the text editor.

Here is an example, note that here the IP has the format 192.168.1.X


[connection]

id=my-ethernet

type=ethernet

interface-name=eth0

permissions=

secondaries=



[ethernet]

mac-address-blacklist=



[ipv4]

#This is the important line
address1=192.168.1.100/24,192.168.1.1 

dns=8.8.8.8;8.8.4.4;

dns-search=

method=manual



[ipv6]

addr-gen-mode=stable-privacy

dns-search=

method=auto

Port Forwarding

Since we are building a server, we need to allow people to be able to connect to our Raspberry Pi. Normally a home router will block all ingoing connections (connections that are not initiated from a device inside the local network), thus we need to create a rule and tell the router that each connection that is made to a specific port should be forwarded to the same port of the Raspberry.

This is because when someone attempts to connect to our IP, in essence, he tries to connect to our router since it functions as the gateway that stands between the Internet and our local network. Thus we want to tell the router that any time someone tries to continue to the ports that we will specify, in essence, he wants to connect to our server, thus the router must forward the connection to the Raspberry pi.

In other words, we need to forward ports 80 and 443 to the Raspberry pi.

Visit portforward
Find your router
Follow instructions and forward the ports to the device's IP

We are almost there, let's deploy the device.

Deploy

Now that we have everything configured, let's start deploying the various components.

Deploy the project to the device

Open balenaCloud and create a new application for Raspberry pi 4. You can name it whatever you want. We will assume that you name it bananas.
Download an development image for that application. Don't bother filling in your wifi credentials, we will be using ethernet as wifi is not very reliable for a webserver.
Burn the image using the best image format app in the world, balena-etcher.
Pull out the sd card and re-insert it in the computer. Copy the file named static-ip we created earlier into the system-connections directory of the sd card.
Insert the card into the Raspberry pi and connect it to power.
Go to the balena-nginx directory, where your docker-compose.yaml is located, and push the project


balena push bananas

Generating the SSL certificate

To generate the SSL certificate, we don't have to do anything.

The server will detect the absence of the certificates and will run the certbot service to register your website. Afterward, it will simply start the server and will serve your website.

The certificates will be saved into a persistent directory with a functionality called named volumes. This enables the device to persist your certificates (or any data in that directory)

Updating your certificates

You are very organized and want to plan ahead? Sure, cerbot will email you a couple of days before your certificates are invalidated so you can renew.

When you receive that e-mail, you only need to:

Log in to balena dashboard
Go to device summary
Open ssh session to the nginx container
Type certbot renew
Done

Push new content to the website

All we have to do in order to push new content to the website is to update the source files in the directory from which we have configured nginx to serve our website. In our case /usr/share/nginx/html/.ping

To do that, we have a script inside the nginx container, which downloads the latest version of our website from the GitHub repository we have defined.

To run the script:

SSH into nginx container, preferably using balena dashboard
/update-blog.sh

But, what happens if we want to add new content to our website?

We have to upload the new source files into our GitHub Repository and then we have to run the update-blog.sh script in order to download the new version in the server.

So, in order to push new content to the website:

Change the website's source files
Upload the new files to the Repository and add a commit message to describe the changes
From balena dashboard, ssh into the nginx container and run: /update-blog.sh

Push new content to the website - For advanced users

The whole process has be automated, and you can simply run the script deploy-dev-loca.sh from the local directory of the project, like this:

./deploy-dev-local.sh "<commit message>"

The field <commit message> must be replaced with a commit message for the addition to the GitHub repository, just like you did when you uploaded the files using the website of GitHub.

Before you can you can use the script, you have to open the file using your favorit text editor and replace the following fields:

# If you use 
export J_OUTPUT= Absolute path to the directory of the website's source files

export REPO = Absolute path to the directory of the website's source files

export DEV_UUID= UUID of the device, can be found from the device's dashboard