DEV Community: R. Tyler Croy

Accessing Handlebars variables in an outer scope

R. Tyler Croy — Thu, 08 Jul 2021 16:16:26 +0000

This weekend I learned some unfamiliar behaviors with the way Handlebars handles nested variable scopes. I typically use Handlebars via the handlebars-rust implementation which aims to maintain nearly one to one compatibility with the JavaScript implementation. They have block scope helpers such as #each and #with, both of which create an inner scope for variable resolution. Unfortunately, the syntax can be quite unintuitive for accessing outer scope once in those nested scopes.

Handlebars is a largely declarative templating syntax which uses curlybraces such as {{var}} for variable and helper substitution. The #each helper is important for loops, imagine the following data structure:

{
  "repos" : [
    {
      "name" : "otto"
    },
    {
      "name" : "l4bsd"
    }
  ],
  "mood" : "cool"
}

This could be rendered into a list on a page via:

<ul>{% raw %}
    {{#each data.repos}}
        <li>{{name}}</li>
    {{/each}}{% endraw %}
</ul>

Inside the #each block the values of the indexed object become the scope for variable resolution, such that {{name}} actually refers to data.repos[i].name. This presents problems when the template must refer to outer scope variables, such as mood. In the Rust implementation this variable resolution can be accomplished through a path traversal style syntax such as:

<ul>{% raw %}
    {{#each data.repos}}
        <li>{{name}} is {{../data.mood}}</li>
    {{/each}}{% endraw %}
</ul>

The ../data.mood is all that's needed to refer to the variable in the outer scope of variables. Not what I expected at all, and the only reason I found it was because I found an old issue which alluded to the syntax and gave it a try.

Increasing the density of the home lab with FreeBSD Jails

R. Tyler Croy — Sun, 06 Jun 2021 18:01:25 +0000

Investing the time to learn FreeBSD jails has led to a dramatic increase in the number of services I run in my "home lab." Jails, which I have written about before, are effectively a lightweight quasi-virtualization technique which I use to create multiple software-defined networks to segment workloads. Jails have allowed me to change my "home lab" dramatically, allowing me to reduce the number of machines and increase hardware utilization. For now, the days of stacking machines, dangling Raspberry Pis, or hiding laptops on the shelf are all gone. Almost all my needs have been consolidated into a single FreeBSD machine running on a 4 year old used workstation.

I have attempted to consolidate the home lab before, but not with this level of success. There was the time I tried deploying a single-node Kubernetes cluster, which worked for a time until some upgrade caused the software-defined networking to break in a way I wasn't interested in debugging in my free time. Following that I tried to go "old school" and started spinning up libvirt-based virtual machines which worked well for a long time. The major downside of that approach is that I simply wasn't able to get much density because of the significant overhead for each virtual machine. At some point you run out of memory to commit to each VM.

Converting virtual machines to FreeBSD Jails took a few hours over a weekend, and since then the quantity of what is running has jumped dramatically. Originally I was running:

Nextcloud (Apache)
MySQl
Gopher server
Tor relay
Gitea

I now also run:

Elasticsearch
Graylog
Icinga
Icecast
MongoDB
Nginx
Peertube
PostgreSQL, which replaced MySQL for Nextcloud and is now running for multiple services.
Redis
Plus some personal web apps

To accomplish this, I use ZFS and Jails heavily. All the jails run on a striped and mirrored ZFS disk array which allows for better performance and redundancy than my previous incantation. These services are low-utilization which allows them to cooperate effectively within this machine's 4 cores and 16GB of RAM.

Benefits

The higher utilization of this single machine, which is always-on, has a better power consumption profile than running multiple machines. More power efficiency wasn't my original motivation however, I was much more interested in some of the system administration benefits of bringing these services "under one roof."

Backups are much easier than before. I'm using some ZFS scripts to perform automatic snapshots on daily/weekly/monthly basis. Separately from those, I regularly push backups off-site and that's trivial because at the "host" machine level, not the jail level, I can access all the filesystems at once.

Security is much easier as well since Jails are obviously providing a level of isolation. On top of that, I am using the vnet functionality in FreeBSD to provide additional network segmentation, something I never did with physical or virtual machines.

Management is pretty straightforward with all these services running in Jails. I typically do all my administration from the host rather than the jail level. As such I can pretty easily shuffle configuration files around. Unfortunately to date I haven't found a good configuration management tool, including my own tinkering that provides a Jails-aware set of tools for more automation.

Downsides

The most notable downside to this approach is that a hardware failure could take me offline for a bit. These services are all dependent on a single power supply (PSU) and CPU. As such a failure of either would require me to keep all these services offline until that part could be replaced. For a home lab setup, I'm not expecting to support a specific SLA, so I'm quite comfortable with this risk.

I can imagine some security arguments that could be made, but frankly I think this Jails approach is much better secured than my previous home lab setups.

At present this machine has just under 10GB of RAM in use and a load average that floats between 1.0 and 2.0. Despite all the services that are running, it still uses less than my workstation with a few browsers open. To deploy your own "home lab" set up in this way you absolutely do not need a high powered machine. I would argue that a ZFS-based disk array is likely more important, after all most home lab tasks, that aren't video-encoding, tend to be more disk I/O heavy rather than memory/CPU heavy.

This path isn't quite entry-level simple, but if you have some systems administration experience, I think FreeBSD with ZFS and Jails is worth considering for your home or office lab.

Converting XML to JSON in Rust

R. Tyler Croy — Sun, 23 May 2021 17:43:34 +0000

I generally default to using JSON for data interchange but there are still a myriad of formats of XML out there, for which I have created the xmltojson crate. I originally wrote this one night to help me get an XML dataset into JSON so that I could use PostgreSQL's JSONB column type, but I only recently published it to crates.io since it may be useful for others.

Cargo.toml

[dependencies]
xmltojson = "0"

The xmltojson crate implements Stefan Goessner’s xml2json which results in a fairly straightforward conversion of XML data structure into JSON. Take the following XML for example:

<ol class="xoxo">
    <li>Subject 1
        <ol>
            <li>subpoint a</li>
            <li>subpoint b</li>
        </ol>
    </li>
    <li>
        <span>Subject 2</span>
        <ol compact="compact">
            <li>subpoint c</li>
            <li>subpoint d</li>
        </ol>
    </li>
</ol>

This XML structured will be rapidly converted into the following JSON, with
attributes and children encoded into the structure:

{
    "ol": {
        "@class":"xoxo",
        "li": [
            {
                "#text":"Subject 1",
                "ol":{
                    "li":[
                        "subpoint a",
                        "subpoint b"
                    ]
                }
            },
            {
                "span":"Subject 2",
                "ol": {
                    "@compact":"compact",
                    "li": [
                        "subpoint c",
                        "subpoint d"
                    ]
                }
            }
        ]
    }
}

There are some oddities with the JSON encoding of XML, particularly around CDATA, but overall I have been quite pleased turning XML into JSON which I can more easily query with jq, PostgreSQL, or even ingest into Elasticsearch.

If you happen to find any bugs, please submit pull requests on GitHub

Dynamically forwarding ports within an existing SSH connection

R. Tyler Croy — Sun, 16 May 2021 20:50:45 +0000

Working over SSH on a big beefy remote machine is a great way to extend the life of any laptop, but presents challenges when developing network-based services. Fortunately OpenSSH has a "port forwarding" feature which has been around for a number of years. Port forwarding allows the user to tunnel a port from the remote machine back to their local machine, in essence allowing you to access a remote service bound to port 8000 on your own localhost:8000. When I first learned about this, I would fiddle around with my ssh invocations or hardcode a list of potential ports forwarded in my ~/.ssh/config. If I was working on a new service that needed a port not yet forwarded, I would disconnect, add it to the list of ports in my config file, and then reconnect. That was until my pal nibz (nibalizer) showed me how to dynamically add port forwards to an already existing session.

OpenSSH provides a number of escape characters that can be used on a running connection, one of these is ~C which will open up a little command line interface:

❯
ssh> ?
Commands:
      -L[bind_address:]port:host:hostport    Request local forward
      -R[bind_address:]port:host:hostport    Request remote forward
      -D[bind_address:]port                  Request dynamic forward
      -KL[bind_address:]port                 Cancel local forward
      -KR[bind_address:]port                 Cancel remote forward
      -KD[bind_address:]port                 Cancel dynamic forward

From this simple command line you can add or remove local, remote, and dynamic port forwards! Pretty snazzy! When I was first learning how to use these escape characters I had a difficult time getting the key-combination correct, frequently I would somehow screw it up and just send ~C to the shell:

❯ ~C
zsh: no such user or named directory:

The approach that I have found most reliable is to hold my left shift key down while I hit ~ and C in sequence. In essence, my left pinky finger keeps shift depressed while my middle finger and then my pointer finger consecutively hit the two keys. I share this because when I first started to use these escape sequences I could never get the timing/cadence correct the first time and it felt like I was wasting more time than it would take to just disconnect and
reconnect.

Another tip is to make sure you have detached from any active tmux sessions, as tmux has it's own escape sequences and key bindings that you likely want to avoid triggering.

Once you get the hang of it, you can dynamically allocate new ports for whatever service you're developing and then easily use your local browser/client to access the service under development. Much better!

For more escape characters, refer to the ssh(1) manpage via man ssh which has a section dedicated to these magic key combos.

ESCAPE CHARACTERS
     When a pseudo-terminal has been requested, ssh supports a number of functions through the use of an escape character.

     A single tilde character can be sent as ~~ or by following the tilde by a character other than those described below.  The escape
     character must always follow a newline to be interpreted as special.  The escape character can be changed in configuration files us-
     ing the EscapeChar configuration directive or on the command line by the -e option.

     The supported escapes (assuming the default ‘~’) are:

     ~.      Disconnect.

     ~^Z     Background ssh.

     ~#      List forwarded connections.

     ~&      Background ssh at logout when waiting for forwarded connection / X11 sessions to terminate.

     ~?      Display a list of escape characters.

     ~B      Send a BREAK to the remote system (only useful if the peer supports it).

     ~C      Open command line.  Currently this allows the addition of port forwardings using the -L, -R and -D options (see above).  It
             also allows the cancellation of existing port-forwardings with -KL[bind_address:]port for local, -KR[bind_address:]port for
             remote and -KD[bind_address:]port for dynamic port-forwardings.  !command allows the user to execute a local command if the
             PermitLocalCommand option is enabled in ssh_config(5).  Basic help is available, using the -h option.

     ~R      Request rekeying of the connection (only useful if the peer supports it).

     ~V      Decrease the verbosity (LogLevel) when errors are being written to stderr.

     ~v      Increase the verbosity (LogLevel) when errors are being written to stderr.

Generating pre-signed S3 URLs in Rust

R. Tyler Croy — Sun, 16 May 2021 14:57:12 +0000

Creating Pre-signed S3 URLs in Rust took me a little more brainpower than I had anticipated, so I thought I would share how to generate them using Rusoto. Pre-signed URLs allow the creation of purpose built URLs for fetching or uploading objects to S3, and can be especially useful when granting access to S3 objects to mobile or web clients. In my use-case, I wanted the clients of my web service to be able to access some specific objects from a bucket.

Rusoto supports the creation of pre-signed URLs via the PreSignedRequest which is implemented for GetObjectRequest, PutObjectRequest, etc. The trait exposes a simple method get_presigned_url which returns a String with all the query parameters to allow for a pre-signed request. Unfortunately however, these GetObjectRequest structs don't really blend easily with an existing S3Client and need to be constructed with the appropriate region and credentials whenever you want to use them.

Starting with the region, I re-use some code we have in delta-rs for identifying the region in a way that allows testing with localstack or minio via the AWS_ENDPOINT_URL environment variable:

use rusoto_core::Region;

let region = if let Ok(url) = std::env::var("AWS_ENDPOINT_URL") {
    Region::Custom {
        name: std::env::var("AWS_REGION").unwrap_or_else(|_| "custom".to_string()),
        endpoint: url,
    }
} else {
    Region::default()
};

For most users, this code doesn't really do much, but if you've got a custom AWS_REGION or AWS_ENDPOINT_URL, you need to properly construct a custom Region in order for Rusoto to work.

The next important argument that get_presigned_url requires is an AwsCredentials provider, which I was originally quite worried about hacking into place. Once again I went looking at the delta-rs codebase for inspiration and noticed our use of ChainProvider which tries its best to find the right AWS credentials given the user's environment:

use rusoto_credential::ChainProvider;
use rusoto_credential::ProvideAwsCredentials;

let provider = ChainProvider::new();
let credentials = provider.credentials().await?;

With those two pieces in place, I could finally construct the URL!

use rusoto_s3::GetObjectRequest;
use rusoto_s3::util::{PreSignedRequest, PreSignedRequestOption};

let options = PreSignedRequestOption {
    expires_in: std::time::Duration::from_secs(300),
};
let req = GetObjectRequest {
    bucket: "my-bucket".to_string(),
    key: "secret.txt".to_string(),
    ..Default::default()
};
let url = req.get_presigned_url(&region, &credentials, &options);

Of course, in your application you might find the structure of managing a shared credentials provider or region to change the structure of the code. However you manage them, as long as you can plug a reference to either into the get_presigned_url function, you can generate useful pre-signed URLs for S3, Minio, etc.