DEV Community: Andrew B. Collier

HCRIS Field Labels

Andrew B. Collier — Tue, 19 Oct 2021 00:00:00 +0000

Fathom Data has been doing a lot of work with the HCRIS (Healthcare Cost Report Information System) data. The underlying reports are submitted as a spreadsheet with multiple sheets. The data are then extracted and recorded in a simple tabular format, with each field linked to a worksheet code (wksht_cd), column number (clmn_num) and line number (clmn_num). These three keys are then mapped to a single compound key. The resulting data look something like this:

       key value
--------------------------
   a_c1_30 8960498
   a_c2_30 2559809
   a_c6_30 -1581948
  a_c1_200 45448537
  a_c2_200 100473288
  a_c6_200 -23947062
 a7_1_c1_1 2696198
 a7_1_c1_6 87655935
 a7_1_c2_6 4190056
 a7_1_c1_8 168428817
 a7_1_c2_8 4391574
a7_1_c1_10 168428817
a7_1_c2_10 4391574

Of course, for these data to make any sense at all it’s necessary to map backwards from the compound key to the location within the original spreadsheet. And once you know where the data belong, you’ll also want to know what the data mean.

Fortunately this relationship is documented and available as a PDF file.

This is useful, but for analytical purposes a PDF is not ideal. We used the {pdftools} package to scrape the text from the PDF and, after a significant bit of wrangling ended up with a nice clean CSV file.

head(hcris_fields)

# A tibble: 6 × 6
  key type wksht_cd clmn_num line_num label                 
  <chr> <chr> <chr> <chr> <chr> <chr>                 
1 a7_1_c1_1 NUMERIC A700001 00100 00100 Land                  
2 a7_1_c1_10 NUMERIC A700001 00100 01000 Total                 
3 a7_1_c1_2 NUMERIC A700001 00100 00200 Land Improvements     
4 a7_1_c1_3 NUMERIC A700001 00100 00300 Buildings and Fixtures
5 a7_1_c1_4 NUMERIC A700001 00100 00400 Building Improvements 
6 a7_1_c1_5 NUMERIC A700001 00100 00500 Fixed Equipment

Resources

Here are the artefacts of this analysis:

Fathoming Email Headers

Andrew B. Collier — Mon, 11 Oct 2021 00:00:00 +0000

If you pull back the curtain and take a look at what a naked email looks like, then you might be astonished. For a start, despite the fact that an email can embed binary data, the message itself is generally transformed to 7 bit ASCII (the first 128 symbols in ASCII). This means that you can easily read the contents (although not all of it might make much sense). At the top of the message are the headers. Your email client will normally extract some pertinent information from these headers (like the contents from the From, To and Date fields) and display it at above the message body. But there’s a lot more information embedded in the headers. In this post we’re going to take a look at what some of those headers mean and what information can be gleaned from them.

Typical Headers

Below is the raw content of a simple (text) email sent from Alice (alice@gmail.com) to Bob (bob@yahoo.com).

Delivered-To: bob@yahoo.com
Received: from 10.214.167.142
 by atlas103.free.mail.gq1.yahoo.com with HTTPS; Fri, 8 Oct 2021 03:34:29 +0000
Return-Path: <alice@gmail.com>
X-Originating-Ip: [209.85.221.51]
Authentication-Results: atlas103.free.mail.gq1.yahoo.com;
 dkim=pass header.i=@gmail-com.20210112.gappssmtp.com header.s=20210112;
 spf=none smtp.mailfrom=gmail.com;
 dmarc=unknown header.from=gmail.com;
X-Apparently-To: bob@yahoo.com; Fri, 8 Oct 2021 03:34:29 +0000
Received: from 209.85.221.51 (EHLO mail-wr1-f51.google.com)
 by 10.214.167.142 with SMTPs
 (version=TLS1_2 cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256);
 Fri, 08 Oct 2021 03:34:29 +0000
Received: by mail-wr1-f51.google.com with SMTP id o20so25174853wro.3
        for <bob@yahoo.com>; Thu, 07 Oct 2021 20:34:29 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail-com.20210112.gappssmtp.com; s=20210112;
        h=message-id📅mime-version:to:from:subject:content-disposition
         :content-transfer-encoding:content-md5;
        bh=krdyOAo/jiepPlfm3uymwB2gf1qtzni7L7sg3hCmaSU=;
        b=8J0ymoL07NMNgk/0NXGCujWtAZ62KdnEk3HwxZpQS99M4PD4/MKKYhjrJxzt5QJGUq
         erS+1nXOeHZD5k7IVlUo7rDJZbDQdt4FFh1wOEaWc8CUPqBu3hJDSgDdWmQRVlsntnnc
         CB6tqF/VC3C4jdoBXX39npp+FFJSBNWcVsZLHdqj1dxhHWbIed3Q98Lfkh+rrb7xHBy4
         cKzdloNNisVPRKQXnNENWRxAF+22fS6DuvfsFyZLctlvgRg8WXGDQACt6WR4prfuV1R3
         thP7NoM7DIIYn3PnepZ3zAbN8P5GG+VOqv64L3sJNUkxrEcNczLdqDDwnyhfUtPKB0wP
         hrig==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=1e100.net; s=20210112;
        h=x-gm-message-state:message-id:date:mime-version:to:from:subject
         :content-disposition:content-transfer-encoding:content-md5;
        bh=krdyOAo/jiepPlfm3uymwB2gf1qtzni7L7sg3hCmaSU=;
        b=ul2T8UXULfKNYI7WHtEG1+zImSks02kwY4jormij3ejAZpdUnYQeNuiBFWAZV+2SWU
         qfLLEEPLcU4Wp9CwX+CbdOZZVM2UKo00EDUjJ3eyOnv0MQy0aBajV27x6L5gKWcURuP4
         rPwGEOInoRE3scYbEwmimebQC8xNQD2kCVJH2HfEII3g5L5U8EVk4UH2HxT7LOwwYfx6
         Gwlhqf4z7xdJc56ywP7UjRXepIEKo2GRwUxs7BrM/v22380Yge6enehyfJCvRmQe+34z
         6Qh0eowKDRbKawIl9RHE6/h1OWBw9mC3I/VzHoXV7+DoXUUYJ883TuTSR47cQ7sqx3+Z
         24KQ==
Received: from allieyoo (host-92-12-241-137.as13285.net. [92.12.241.137])
        by smtp.gmail.com with ESMTPSA id u5sm1069389wrg.57.2021.10.07.20.34.28
        for <bob@yahoo.com>
        (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256);
        Thu, 07 Oct 2021 20:34:28 -0700 (PDT)
Message-ID: <615fbc44.1c69fb81.55d71.5951@mx.google.com>
Date: Thu, 07 Oct 2021 20:34:28 -0700 (PDT)
X-Google-Original-Date: Fri, 08 Oct 2021 03:34:27 GMT
From: alice@gmail.com
To: bob@yahoo.com
Subject: Hello
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Disposition: inline
Content-Transfer-Encoding: 7bit
Content-MD5: ZajifYh5KDgxtmS9i38K1A==
Content-Length: 14
Content-Language: en-GB

Hello, World!

If you skimmed through the raw message contents then (hopefully before your eyes glazed over) you would have noted that the headers consist of key-value pairs. For example, Date: Thu, 07 Oct 2021 20:34:28 -0700 (PDT) has key Date and value Thu, 07 Oct 2021 20:34:28 -0700 (PDT).

You might also have noticed that a number of the header keys start with X-. These are custom headers inserted by the various processes that handle the mail while it’s being delivered.

Original Message Content

For reference, here is the content of the message that was actually sent by Alice.

Date: Fri, 08 Oct 2021 03:34:27 GMT
From: alice@gmail.com
To: bob@yahoo.com
Subject: Hello
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Disposition: inline
Content-Transfer-Encoding: 7bit
Content-MD5: ZajifYh5KDgxtmS9i38K1A==

Hello, World!

The message was generated and sent using the {emayili} package. Clearly a lot of headers are introduced in the process of delivering the message!

Number of Headers

You might have noticed that some of the header records in the example above occurred more than once. For example, the Return-Path field. This is not uncommon. However, there are some rules about which headers may be repeated. These rules also indicates which headers are optional and which are mandatory. The table below (extracted from RFC 5322) indicates the limits on the number of times each type of header should appear in an email. The range of headers included in the table is only a subset of all possible headers. But it gives a fair idea of what the rules are. For example, the From field must be specified, as must the Date field. The To, Cc and Bcc fields are all optional though. So, in principle, you can create a valid email with no recipients.

Source Header Fields

First we’ll take a look at the header fields which are part of the original (source) message as sent. These fields are all populated by the email client.

`Date`

The Date field specifies the date and time at which the message was sent.

You might note that the date specified in the source message was Fri, 08 Oct 2021 03:34:27 GMT, while the date in the delivered message was Thu, 07 Oct 2021 20:34:28 -0700 (PDT).

The format of the date and time was originally defined in RFC 822 Section 5 and subsequently clarified in RFC 2822 Section 3.3. However you will find that there is quite a variety in the formats used in practice.

The custom X-Google-Original-Date field retains the date and time in their original source format.

`From` and `To`

The From and To fields specify the email addresses of the sender and recipient of the message. The format for the addresses is detaied in RFC 822 Section 6.

In addition to these fields you can also have Cc (carbon copy.) and Bcc (blind carbon copy.) fields, which specify additional recipients who are not the main intended recipient. The details of those listed under Bcc are not visible to any of the recipients.

`Subject`

The Subject field doesn’t really require much explanation: it tells you what the message is about.

`MIME-Version`

Multipurpose Internet Mail Extensions (MIME) is a format which allows email messages to contain content beyond mere text. MIME is described in great detail in RFC 1341. The version of MIME is typically 1.0 because it appears that’s the only version available!

`Content-Type`

The Content-Type field specifies the media type (or MIME type) of data in the message. The type is specified as two-part identifier with <type>/<subtype>. For example, text/plain or text/html for plain text or HTML content, and image/png for a PNG image. An extensive list of a wide variety of media types can be found here.

The media type is often followed by one or more parameters. For example, in Alice’s email the Content-Type field is text/plain; charset=utf-8; format=flowed, where the charset and format parameters indicate UTF-8 encoding of the text and flowed format (lines are not wrapped).

In a multi-part message (for example, a message with both plain and HTML test as well as attachments) there will be multiple Content-Type fields, one for each part of the message.

`Content-Disposition`

The Content-Disposition header indicates how content is to be displayed, and is generally either inline or attachment. RFC 2183 is an extended discussion of this field. The range of permitted values is shown in the table below.

html {
font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, Oxygen, Ubuntu, Cantarell, 'Helvetica Neue', 'Fira Sans', 'Droid Sans', Arial, sans-serif;
}

wkszanyzfl .gt_table {

display: table;
border-collapse: collapse;
margin-left: 0;
margin-right: auto;
color: #333333;
font-size: 16px;
font-weight: normal;
font-style: normal;
background-color: #FFFFFF;
width: auto;
border-top-style: solid;
border-top-width: 2px;
border-top-color: #A8A8A8;
border-right-style: none;
border-right-width: 2px;
border-right-color: #D3D3D3;
border-bottom-style: solid;
border-bottom-width: 2px;
border-bottom-color: #A8A8A8;
border-left-style: none;
border-left-width: 2px;
border-left-color: #D3D3D3;
}

wkszanyzfl .gt_heading {

background-color: #FFFFFF;
text-align: center;
border-bottom-color: #FFFFFF;
border-left-style: none;
border-left-width: 1px;
border-left-color: #D3D3D3;
border-right-style: none;
border-right-width: 1px;
border-right-color: #D3D3D3;
}

wkszanyzfl .gt_title {

color: #333333;
font-size: 125%;
font-weight: initial;
padding-top: 4px;
padding-bottom: 4px;
border-bottom-color: #FFFFFF;
border-bottom-width: 0;
}

wkszanyzfl .gt_subtitle {

color: #333333;
font-size: 85%;
font-weight: initial;
padding-top: 0;
padding-bottom: 6px;
border-top-color: #FFFFFF;
border-top-width: 0;
}

wkszanyzfl .gt_bottom_border {

border-bottom-style: solid;
border-bottom-width: 2px;
border-bottom-color: #D3D3D3;
}

wkszanyzfl .gt_col_headings {

border-top-style: solid;
border-top-width: 2px;
border-top-color: #D3D3D3;
border-bottom-style: solid;
border-bottom-width: 2px;
border-bottom-color: #D3D3D3;
border-left-style: none;
border-left-width: 1px;
border-left-color: #D3D3D3;
border-right-style: none;
border-right-width: 1px;
border-right-color: #D3D3D3;
}

wkszanyzfl .gt_col_heading {

color: #333333;
background-color: #FFFFFF;
font-size: 100%;
font-weight: normal;
text-transform: inherit;
border-left-style: none;
border-left-width: 1px;
border-left-color: #D3D3D3;
border-right-style: none;
border-right-width: 1px;
border-right-color: #D3D3D3;
vertical-align: bottom;
padding-top: 5px;
padding-bottom: 6px;
padding-left: 5px;
padding-right: 5px;
overflow-x: hidden;
}

wkszanyzfl .gt_column_spanner_outer {

color: #333333;
background-color: #FFFFFF;
font-size: 100%;
font-weight: normal;
text-transform: inherit;
padding-top: 0;
padding-bottom: 0;
padding-left: 4px;
padding-right: 4px;
}

wkszanyzfl .gt_column_spanner_outer:first-child {

padding-left: 0;
}

wkszanyzfl .gt_column_spanner_outer:last-child {

padding-right: 0;
}

wkszanyzfl .gt_column_spanner {

border-bottom-style: solid;
border-bottom-width: 2px;
border-bottom-color: #D3D3D3;
vertical-align: bottom;
padding-top: 5px;
padding-bottom: 5px;
overflow-x: hidden;
display: inline-block;
width: 100%;
}

wkszanyzfl .gt_group_heading {

padding: 8px;
color: #333333;
background-color: #FFFFFF;
font-size: 100%;
font-weight: initial;
text-transform: inherit;
border-top-style: solid;
border-top-width: 2px;
border-top-color: #D3D3D3;
border-bottom-style: solid;
border-bottom-width: 2px;
border-bottom-color: #D3D3D3;
border-left-style: none;
border-left-width: 1px;
border-left-color: #D3D3D3;
border-right-style: none;
border-right-width: 1px;
border-right-color: #D3D3D3;
vertical-align: middle;
}

wkszanyzfl .gt_empty_group_heading {

padding: 0.5px;
color: #333333;
background-color: #FFFFFF;
font-size: 100%;
font-weight: initial;
border-top-style: solid;
border-top-width: 2px;
border-top-color: #D3D3D3;
border-bottom-style: solid;
border-bottom-width: 2px;
border-bottom-color: #D3D3D3;
vertical-align: middle;
}

wkszanyzfl .gt_from_md > :first-child {

margin-top: 0;
}

wkszanyzfl .gt_from_md > :last-child {

margin-bottom: 0;
}

wkszanyzfl .gt_row {

padding-top: 8px;
padding-bottom: 8px;
padding-left: 5px;
padding-right: 5px;
margin: 10px;
border-top-style: solid;
border-top-width: 1px;
border-top-color: #D3D3D3;
border-left-style: none;
border-left-width: 1px;
border-left-color: #D3D3D3;
border-right-style: none;
border-right-width: 1px;
border-right-color: #D3D3D3;
vertical-align: middle;
overflow-x: hidden;
}

wkszanyzfl .gt_stub {

color: #333333;
background-color: #FFFFFF;
font-size: 100%;
font-weight: initial;
text-transform: inherit;
border-right-style: solid;
border-right-width: 2px;
border-right-color: #D3D3D3;
padding-left: 12px;
}

wkszanyzfl .gt_summary_row {

color: #333333;
background-color: #FFFFFF;
text-transform: inherit;
padding-top: 8px;
padding-bottom: 8px;
padding-left: 5px;
padding-right: 5px;
}

wkszanyzfl .gt_first_summary_row {

padding-top: 8px;
padding-bottom: 8px;
padding-left: 5px;
padding-right: 5px;
border-top-style: solid;
border-top-width: 2px;
border-top-color: #D3D3D3;
}

wkszanyzfl .gt_grand_summary_row {

color: #333333;
background-color: #FFFFFF;
text-transform: inherit;
padding-top: 8px;
padding-bottom: 8px;
padding-left: 5px;
padding-right: 5px;
}

wkszanyzfl .gt_first_grand_summary_row {

padding-top: 8px;
padding-bottom: 8px;
padding-left: 5px;
padding-right: 5px;
border-top-style: double;
border-top-width: 6px;
border-top-color: #D3D3D3;
}

wkszanyzfl .gt_striped {

background-color: rgba(128, 128, 128, 0.05);
}

wkszanyzfl .gt_table_body {

border-top-style: solid;
border-top-width: 2px;
border-top-color: #D3D3D3;
border-bottom-style: solid;
border-bottom-width: 2px;
border-bottom-color: #D3D3D3;
}

wkszanyzfl .gt_footnotes {

color: #333333;
background-color: #FFFFFF;
border-bottom-style: none;
border-bottom-width: 2px;
border-bottom-color: #D3D3D3;
border-left-style: none;
border-left-width: 2px;
border-left-color: #D3D3D3;
border-right-style: none;
border-right-width: 2px;
border-right-color: #D3D3D3;
}

wkszanyzfl .gt_footnote {

margin: 0px;
font-size: 90%;
padding: 4px;
}

wkszanyzfl .gt_sourcenotes {

wkszanyzfl .gt_sourcenote {

font-size: 90%;
padding: 4px;
}

wkszanyzfl .gt_left {

text-align: left;
}

wkszanyzfl .gt_center {

text-align: center;
}

wkszanyzfl .gt_right {

text-align: right;
font-variant-numeric: tabular-nums;
}

wkszanyzfl .gt_font_normal {

font-weight: normal;
}

wkszanyzfl .gt_font_bold {

font-weight: bold;
}

wkszanyzfl .gt_font_italic {

font-style: italic;
}

wkszanyzfl .gt_super {

font-size: 65%;
}

wkszanyzfl .gt_footnote_marks {

font-style: italic;
font-weight: normal;
font-size: 65%;
}

Content-Disposition Values
value
---

inline

| RFC 2183 |
|

attachment

| RFC 2183 |
|

form-data

| RFC 7578 |
|

signal

| RFC 3204 |
|

alert

| RFC 3261 |
|

icon

| RFC 3261 |
|

render

| RFC 3261 |
|

recipient-list-history

| RFC 5364 |
|

session

| RFC 3261 |
|

aib

| RFC 3893 |
|

early-session

| RFC 3959 |
|

recipient-list

| RFC 5363 |
|

notification

| RFC 5438 |
|

by-reference

| RFC 5621 |
|

info-package

| RFC 6086 |
|

recording-session

| RFC 7866 |

There are also various parameters which can be specified for this field, with options extracted from the table below.

html { font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, Oxygen, Ubuntu, Cantarell, 'Helvetica Neue', 'Fira Sans', 'Droid Sans', Arial, sans-serif; } #ejufukiovl .gt_table { display: table; border-collapse: collapse; margin-left: 0; margin-right: auto; color: #333333; font-size: 16px; font-weight: normal; font-style: normal; background-color: #FFFFFF; width: auto; border-top-style: solid; border-top-width: 2px; border-top-color: #A8A8A8; border-right-style: none; border-right-width: 2px; border-right-color: #D3D3D3; border-bottom-style: solid; border-bottom-width: 2px; border-bottom-color: #A8A8A8; border-left-style: none; border-left-width: 2px; border-left-color: #D3D3D3; } #ejufukiovl .gt_heading { background-color: #FFFFFF; text-align: center; border-bottom-color: #FFFFFF; border-left-style: none; border-left-width: 1px; border-left-color: #D3D3D3; border-right-style: none; border-right-width: 1px; border-right-color: #D3D3D3; } #ejufukiovl .gt_title { color: #333333; font-size: 125%; font-weight: initial; padding-top: 4px; padding-bottom: 4px; border-bottom-color: #FFFFFF; border-bottom-width: 0; } #ejufukiovl .gt_subtitle { color: #333333; font-size: 85%; font-weight: initial; padding-top: 0; padding-bottom: 6px; border-top-color: #FFFFFF; border-top-width: 0; } #ejufukiovl .gt_bottom_border { border-bottom-style: solid; border-bottom-width: 2px; border-bottom-color: #D3D3D3; } #ejufukiovl .gt_col_headings { border-top-style: solid; border-top-width: 2px; border-top-color: #D3D3D3; border-bottom-style: solid; border-bottom-width: 2px; border-bottom-color: #D3D3D3; border-left-style: none; border-left-width: 1px; border-left-color: #D3D3D3; border-right-style: none; border-right-width: 1px; border-right-color: #D3D3D3; } #ejufukiovl .gt_col_heading { color: #333333; background-color: #FFFFFF; font-size: 100%; font-weight: normal; text-transform: inherit; border-left-style: none; border-left-width: 1px; border-left-color: #D3D3D3; border-right-style: none; border-right-width: 1px; border-right-color: #D3D3D3; vertical-align: bottom; padding-top: 5px; padding-bottom: 6px; padding-left: 5px; padding-right: 5px; overflow-x: hidden; } #ejufukiovl .gt_column_spanner_outer { color: #333333; background-color: #FFFFFF; font-size: 100%; font-weight: normal; text-transform: inherit; padding-top: 0; padding-bottom: 0; padding-left: 4px; padding-right: 4px; } #ejufukiovl .gt_column_spanner_outer:first-child { padding-left: 0; } #ejufukiovl .gt_column_spanner_outer:last-child { padding-right: 0; } #ejufukiovl .gt_column_spanner { border-bottom-style: solid; border-bottom-width: 2px; border-bottom-color: #D3D3D3; vertical-align: bottom; padding-top: 5px; padding-bottom: 5px; overflow-x: hidden; display: inline-block; width: 100%; } #ejufukiovl .gt_group_heading { padding: 8px; color: #333333; background-color: #FFFFFF; font-size: 100%; font-weight: initial; text-transform: inherit; border-top-style: solid; border-top-width: 2px; border-top-color: #D3D3D3; border-bottom-style: solid; border-bottom-width: 2px; border-bottom-color: #D3D3D3; border-left-style: none; border-left-width: 1px; border-left-color: #D3D3D3; border-right-style: none; border-right-width: 1px; border-right-color: #D3D3D3; vertical-align: middle; } #ejufukiovl .gt_empty_group_heading { padding: 0.5px; color: #333333; background-color: #FFFFFF; font-size: 100%; font-weight: initial; border-top-style: solid; border-top-width: 2px; border-top-color: #D3D3D3; border-bottom-style: solid; border-bottom-width: 2px; border-bottom-color: #D3D3D3; vertical-align: middle; } #ejufukiovl .gt_from_md > :first-child { margin-top: 0; } #ejufukiovl .gt_from_md > :last-child { margin-bottom: 0; } #ejufukiovl .gt_row { padding-top: 8px; padding-bottom: 8px; padding-left: 5px; padding-right: 5px; margin: 10px; border-top-style: solid; border-top-width: 1px; border-top-color: #D3D3D3; border-left-style: none; border-left-width: 1px; border-left-color: #D3D3D3; border-right-style: none; border-right-width: 1px; border-right-color: #D3D3D3; vertical-align: middle; overflow-x: hidden; } #ejufukiovl .gt_stub { color: #333333; background-color: #FFFFFF; font-size: 100%; font-weight: initial; text-transform: inherit; border-right-style: solid; border-right-width: 2px; border-right-color: #D3D3D3; padding-left: 12px; } #ejufukiovl .gt_summary_row { color: #333333; background-color: #FFFFFF; text-transform: inherit; padding-top: 8px; padding-bottom: 8px; padding-left: 5px; padding-right: 5px; } #ejufukiovl .gt_first_summary_row { padding-top: 8px; padding-bottom: 8px; padding-left: 5px; padding-right: 5px; border-top-style: solid; border-top-width: 2px; border-top-color: #D3D3D3; } #ejufukiovl .gt_grand_summary_row { color: #333333; background-color: #FFFFFF; text-transform: inherit; padding-top: 8px; padding-bottom: 8px; padding-left: 5px; padding-right: 5px; } #ejufukiovl .gt_first_grand_summary_row { padding-top: 8px; padding-bottom: 8px; padding-left: 5px; padding-right: 5px; border-top-style: double; border-top-width: 6px; border-top-color: #D3D3D3; } #ejufukiovl .gt_striped { background-color: rgba(128, 128, 128, 0.05); } #ejufukiovl .gt_table_body { border-top-style: solid; border-top-width: 2px; border-top-color: #D3D3D3; border-bottom-style: solid; border-bottom-width: 2px; border-bottom-color: #D3D3D3; } #ejufukiovl .gt_footnotes { color: #333333; background-color: #FFFFFF; border-bottom-style: none; border-bottom-width: 2px; border-bottom-color: #D3D3D3; border-left-style: none; border-left-width: 2px; border-left-color: #D3D3D3; border-right-style: none; border-right-width: 2px; border-right-color: #D3D3D3; } #ejufukiovl .gt_footnote { margin: 0px; font-size: 90%; padding: 4px; } #ejufukiovl .gt_sourcenotes { color: #333333; background-color: #FFFFFF; border-bottom-style: none; border-bottom-width: 2px; border-bottom-color: #D3D3D3; border-left-style: none; border-left-width: 2px; border-left-color: #D3D3D3; border-right-style: none; border-right-width: 2px; border-right-color: #D3D3D3; } #ejufukiovl .gt_sourcenote { font-size: 90%; padding: 4px; } #ejufukiovl .gt_left { text-align: left; } #ejufukiovl .gt_center { text-align: center; } #ejufukiovl .gt_right { text-align: right; font-variant-numeric: tabular-nums; } #ejufukiovl .gt_font_normal { font-weight: normal; } #ejufukiovl .gt_font_bold { font-weight: bold; } #ejufukiovl .gt_font_italic { font-style: italic; } #ejufukiovl .gt_super { font-size: 65%; } #ejufukiovl .gt_footnote_marks { font-style: italic; font-weight: normal; font-size: 65%; }

Content-Disposition Parameters
name
---

filename

| RFC 2183 |
|

creation-date

| RFC 2183 |
|

modification-date

| RFC 2183 |
|

read-date

| RFC 2183 |
|

size

| RFC 2183 |
|

name

| RFC 7578 |
|

voice

| RFC 2421 |
|

handling

| RFC 3204 |
|

preview-type

| RFC 7763 |
|

reaction

| RFC 9078 |

`Content-Transfer-Encoding`

The Content-Transfer-Encoding field is document in RFC 2045 Section 6. It specifies the way that the content has been encoded. Why is encoding necessary? Well, RFC 821 which established SMTP specified that data would be represented by 7-bit ASCII characters.

The TCP connection supports the transmission of 8-bit bytes. The SMTP data is 7-bit ASCII characters. Each character is transmitted as an 8-bit byte with the high-order bit cleared to zero. RFC 821

Furthermore, the maximum length of a line was limited to 1000 characters.

The maximum total length of a text line including the CRLF is 1000 characters (but not counting the leading dot duplicated for transparency). RFC 821

The majority of data that you’d want to include in an email doesn’t comply with either of these requirements. So, in order for you to be able to include fancy Unicode characters in your HTML messages and attach cute cat pictures, those data need to be encoded in such a way that they can still be transmitted over SMTP.

The most common values for this field are

7bit (default) — Content is already in 7-bit ASCII (no encoding).
quoted-printable — Quoted-Printable encoding uses the equal sign (=) as an escape character to transform non-ASCII characters into something that can be represnted by ASCII. For example, the character é (e-acute), which is represented in UTF-8 encoding by the bytes 0xC3 0xA9, is encoded as =C3=A9. This encoding also limits line length to 76 characters, where lines are terminated by a soft line break represented by =. Quoted-Printable encoding is most often used to encode text data.
base64 — Base64 encoding encodes data into 6-bit digits which are packed in groups of four, so that 4 Base64 digits represent 24 bits or 3 bytes. Base64 is most often used to encode binary data.

To illustrate the difference between these various coding mechanisms, suppose that you wanted to send the following message in an email:

J’interdis aux marchands de vanter trop leurs marchandises. Car ils se font vite pédagogues et t’enseignent comme but ce qui n’est par essence qu’un moyen, et te trompant ainsi sur la route à suivre les voilà bientôt qui te dégradent, car si leur musique est vulgaire ils te fabriquent pour te la vendre une âme vulgaire.

Using Quoted-Printable encoding the email would look like this:

Content-Type: text/plain; charset=utf-8; format=flowed
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable
Content-MD5: XjpBVdSoL+frxc/IjptLkA==

J'interdis aux marchands de vanter trop leurs marchandises. Car ils se font=
 vite p=C3=A9dagogues et t'enseignent comme but ce qui n'est par essence qu=
'un moyen, et te trompant ainsi sur la route =C3=A0 suivre les voil=C3=A0 b=
ient=C3=B4t qui te d=C3=A9gradent, car si leur musique est vulgaire ils te =
fabriquent pour te la vendre une =C3=A2me vulgaire.

All non-ASCII characters have been encoded and the text has been wrapped to a width of only 76 characters. Most of the content is still legible though. By contrast, Base64 encoding would look like this:

Content-Type: text/plain; charset=utf-8; format=flowed
Content-Disposition: inline
Content-Transfer-Encoding: base64
Content-MD5: XjpBVdSoL+frxc/IjptLkA==

SidpbnRlcmRpcyBhdXggbWFyY2hhbmRzIGRlIHZhbnRlciB0cm9wIGxldXJzIG1hcmNoYW5kaXNl
cy4gQ2FyIGlscyBzZSBmb250IHZpdGUgcMOpZGFnb2d1ZXMgZXQgdCdlbnNlaWduZW50IGNvbW1l
IGJ1dCBjZSBxdWkgbidlc3QgcGFyIGVzc2VuY2UgcXUndW4gbW95ZW4sIGV0IHRlIHRyb21wYW50
IGFpbnNpIHN1ciBsYSByb3V0ZSDDoCBzdWl2cmUgbGVzIHZvaWzDoCBiaWVudMO0dCBxdWkgdGUg
ZMOpZ3JhZGVudCwgY2FyIHNpIGxldXIgbXVzaXF1ZSBlc3QgdnVsZ2FpcmUgaWxzIHRlIGZhYnJp
cXVlbnQgcG91ciB0ZSBsYSB2ZW5kcmUgdW5lIMOibWUgdnVsZ2FpcmUu

Again it’s all ASCII characters with a line width of only 76 characters. But now the content is completely illegible (unless you can Base64 decode in your head!).

`Content-MD5`

MD5 is a hash function that produces 128-bit hash values. For the purposes of email it’s used as a checksum to verify data integrity. The Content-MD5 header field is a Base64 encoded representation of the MD5 has of the contents of the message component.

In the two examples above (for Quoted-Printable and Base64 content encoding) you’ll notice that the value of the Content-MD5 field, XjpBVdSoL+frxc/IjptLkA==, is the same. This is because the contents of the message components is the same despite them being encoded in different ways. Decoding the Base64 yields the raw MD5 hash, rmsg_unicode_md5_hex`. Check this against the results from the MD5 Hash Generator.

Transit Header Fields

Now let’s look at the header fields introduced during delivery.

`Message-ID`

The Message-ID field uniquely identifies a specific message.

Message-ID: <615fbc44.1c69fb81.55d71.5951@mx.google.com>

The format of the identifier is similar to that of an email address, consisting of two components: an unique identifier (for example, 615fbc44.1c69fb81.55d71.5951) and the domain name of the mail server (for example, mx.google.com).

The Message-ID is important for linking messages together into threads, where the In-Reply-To and References headers are used to reference earlier messages via their Message-ID.

`Received`

There are multiple Received entries in the header. These trace the route that the message took from Alice to Bob.

Received: from 10.214.167.142 by atlas103.free.mail.gq1.yahoo.com with HTTPS; Fri, 8 Oct 2021 03:34:29 +0000 Received: from 209.85.221.51 (EHLO mail-wr1-f51.google.com) by 10.214.167.142 with SMTPs (version=TLS1_2 cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256); Fri, 08 Oct 2021 03:34:29 +0000 Received: by mail-wr1-f51.google.com with SMTP id o20so25174853wro.3 for <bob@yahoo.com>; Thu, 07 Oct 2021 20:34:29 -0700 (PDT) Received: from allieyoo (host-92-12-241-137.as13285.net. [92.12.241.137]) by smtp.gmail.com with ESMTPSA id u5sm1069389wrg.57.2021.10.07.20.34.28 for <bob@yahoo.com> (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 07 Oct 2021 20:34:28 -0700 (PDT)

The message was sent initially to the Gmail SMTP server smtp.gmail.com (IP 64.233.184.109). This is the server configured by Alice for her outgoing email. Where is that server? Good question. I used ipgeolocation to find out (slightly abridged output from their API below).

{ "ip": "64.233.184.109", "continent_name": "North America", "country_name": "United States", "state_prov": "California", "district": "Old Mountain View", "city": "Mountain View", "zipcode": "94041-1238", "latitude": "37.39500", "longitude": "-122.08167", "isp": "Google LLC", "organization": "Google LLC", "time_zone": { "name": "America/Los_Angeles", "offset": -8, "is_dst": true, "dst_savings": 1 } }

So Alice’s message went from her computer in the UK to a server in Mountain View, California on the west coast of the USA. That’s quite a big leap! We can use the other Received fields to track its progress from there. Next stop was mail-wr1-f51.google.com (IP 209.85.221.51), also located in Mountain View, California. From there it went to a machine on a private network (IP 10.214.167.142) before finally being delivered to atlas103.free.mail.gq1.yahoo.com, presumably another server located on a private network.

The Received field can also include information on the encryption (see references to TLS) and the protocol (see ESMTPSA, SMTP and SMTPs) being used.

html {
font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, Oxygen, Ubuntu, Cantarell, 'Helvetica Neue', 'Fira Sans', 'Droid Sans', Arial, sans-serif;
}

qiolitmyfl .gt_table {

qiolitmyfl .gt_heading {

qiolitmyfl .gt_title {

color: #333333;
font-size: 125%;
font-weight: initial;
padding-top: 4px;
padding-bottom: 4px;
border-bottom-color: #FFFFFF;
border-bottom-width: 0;
}

qiolitmyfl .gt_subtitle {

color: #333333;
font-size: 85%;
font-weight: initial;
padding-top: 0;
padding-bottom: 6px;
border-top-color: #FFFFFF;
border-top-width: 0;
}

qiolitmyfl .gt_bottom_border {

border-bottom-style: solid;
border-bottom-width: 2px;
border-bottom-color: #D3D3D3;
}

qiolitmyfl .gt_col_headings {

qiolitmyfl .gt_col_heading {

qiolitmyfl .gt_column_spanner_outer {

color: #333333;
background-color: #FFFFFF;
font-size: 100%;
font-weight: normal;
text-transform: inherit;
padding-top: 0;
padding-bottom: 0;
padding-left: 4px;
padding-right: 4px;
}

qiolitmyfl .gt_column_spanner_outer:first-child {

padding-left: 0;
}

qiolitmyfl .gt_column_spanner_outer:last-child {

padding-right: 0;
}

qiolitmyfl .gt_column_spanner {

qiolitmyfl .gt_group_heading {

qiolitmyfl .gt_empty_group_heading {

qiolitmyfl .gt_from_md > :first-child {

margin-top: 0;
}

qiolitmyfl .gt_from_md > :last-child {

margin-bottom: 0;
}

qiolitmyfl .gt_row {

qiolitmyfl .gt_stub {

qiolitmyfl .gt_summary_row {

color: #333333;
background-color: #FFFFFF;
text-transform: inherit;
padding-top: 8px;
padding-bottom: 8px;
padding-left: 5px;
padding-right: 5px;
}

qiolitmyfl .gt_first_summary_row {

padding-top: 8px;
padding-bottom: 8px;
padding-left: 5px;
padding-right: 5px;
border-top-style: solid;
border-top-width: 2px;
border-top-color: #D3D3D3;
}

qiolitmyfl .gt_grand_summary_row {

color: #333333;
background-color: #FFFFFF;
text-transform: inherit;
padding-top: 8px;
padding-bottom: 8px;
padding-left: 5px;
padding-right: 5px;
}

qiolitmyfl .gt_first_grand_summary_row {

padding-top: 8px;
padding-bottom: 8px;
padding-left: 5px;
padding-right: 5px;
border-top-style: double;
border-top-width: 6px;
border-top-color: #D3D3D3;
}

qiolitmyfl .gt_striped {

background-color: rgba(128, 128, 128, 0.05);
}

qiolitmyfl .gt_table_body {

border-top-style: solid;
border-top-width: 2px;
border-top-color: #D3D3D3;
border-bottom-style: solid;
border-bottom-width: 2px;
border-bottom-color: #D3D3D3;
}

qiolitmyfl .gt_footnotes {

qiolitmyfl .gt_footnote {

margin: 0px;
font-size: 90%;
padding: 4px;
}

qiolitmyfl .gt_sourcenotes {

qiolitmyfl .gt_sourcenote {

font-size: 90%;
padding: 4px;
}

qiolitmyfl .gt_left {

text-align: left;
}

qiolitmyfl .gt_center {

text-align: center;
}

qiolitmyfl .gt_right {

text-align: right;
font-variant-numeric: tabular-nums;
}

qiolitmyfl .gt_font_normal {

font-weight: normal;
}

qiolitmyfl .gt_font_bold {

font-weight: bold;
}

qiolitmyfl .gt_font_italic {

font-style: italic;
}

qiolitmyfl .gt_super {

font-size: 65%;
}

qiolitmyfl .gt_footnote_marks {

font-style: italic;
font-weight: normal;
font-size: 65%;
}

Mail Protocol Types
protocol
---
SMTP
ESMTP
ESMTPA
ESMTPS
ESMTPSA
LMTP
LMTPA
LMTPS
LMTPSA
MMS
UTF8SMTP
UTF8SMTPA
UTF8SMTPS
UTF8SMTPSA
UTF8LMTP
UTF8LMTPA
UTF8LMTPS
UTF8LMTPSA

There can also be an X-Received field which is a custom field containing information similar to the Received field.

`X-Originating-Ip`

The X-Originating-Ip field identifies the IP address of the sender. This is relevant, for example, when you send an email using a web email client. In this case the web client communicates with the SMTP server, but the content of this field will be your actual IP address.

`Delivered-To` and `Return-Path`

The Delivered-To field records the email address to which the message was actually delivered. Somewhat surprisingly, this can be different to the addresses specified in the To, Cc or Bcc fields. How does this happen? An email message generally passes through numerous servers and processes between the sender and the recipient. At various stages in this process rules might be applied which modify the delivery of the message. Suppose, for example, that Bob (bob@yahoo.com) is one of many clients with who Alice communicates. If she sends out a general email to all of her clients then she will probably use a mailing list rather than adding the address of each client individually. The message To field would then contain the name of the mailing list, but this would be expanded en route to a list of individual email addresses, and these would appear in the Delivered-To field.

Sometimes an email doesn’t get delivered. Maybe there’s a problem with the recipients email server. Or perhaps the address of the recipient has changed. In this case the email will bounce and a bounce message will be returned which informs the sender that the email did not reach its destination. The Return-Path field specifies the address to which the bounce message should be delivered. If, for example, you’re sending email to a large mailing list and some of the addresses on that list are unreliable then it can be useful to specify a different Return-Path address so that bounce messages don’t end up cluttering your inbox. Although you can specify a value for Return-Path it’s possible that your email server will override this.

`Authentication-Results`

The Authentication-Results header contains information on what authentication methods have been applied to a message. It will often include information on the following protocols, all of which are intended to detected forged sender addresses (also known as “email spoofing”):

DKIM (DomainKeys Identified Mail)
SPF (Sender Policy Framework) and
DMARC (Domain-based Message Authentication, Reporting and Conformance).

Authentication-Results: atlas103.free.mail.gq1.yahoo.com; dkim=pass header.i=@gmail-com.20210112.gappssmtp.com header.s=20210112; spf=none smtp.mailfrom=gmail.com; dmarc=unknown header.from=gmail.com;

In the example above the results for each of these methods are listed. Valid results for each method are:

DKIM — pass, fail or none
SPF — pass, fail, softfail, neutral, none, temperror or permerror
DMARC — pass, fail, bestguesspass, none or unknown.

`DKIM-Signature`

DKIM (DomainKeys Identified Mail) is an authentication protocol. A DKIM Signature verifies the DNS domain of the email sender. The DNS entry for a mail server is associated with a public key, that is published and freely available. The DKIM Signature is a signed with the corresponding private key. Using the public key it’s then possible to establish the validity of the sender’s email address.

DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail-com.20210112.gappssmtp.com; s=20210112; h=message-id📅mime-version:to:from:subject:content-disposition :content-transfer-encoding:content-md5; bh=krdyOAo/jiepPlfm3uymwB2gf1qtzni7L7sg3hCmaSU=; b=8J0ymoL07NMNgk/0NXGCujWtAZ62KdnEk3HwxZpQS99M4PD4/MKKYhjrJxzt5QJGUq erS+1nXOeHZD5k7IVlUo7rDJZbDQdt4FFh1wOEaWc8CUPqBu3hJDSgDdWmQRVlsntnnc CB6tqF/VC3C4jdoBXX39npp+FFJSBNWcVsZLHdqj1dxhHWbIed3Q98Lfkh+rrb7xHBy4 cKzdloNNisVPRKQXnNENWRxAF+22fS6DuvfsFyZLctlvgRg8WXGDQACt6WR4prfuV1R3 thP7NoM7DIIYn3PnepZ3zAbN8P5GG+VOqv64L3sJNUkxrEcNczLdqDDwnyhfUtPKB0wP hrig==

The DKIM Signature field is comprised of a number of elements:

v — the DKIM version
a — the signing algorithm
c — algorithm used to canonicalise the header and body (optional)
d — Signing Domain Identifier (SDID) is the domain used to sign the email
s — a selector
h — list of header fields that were signed
bh — a hash of the message body
b — signature of the header fields and body.

The custom X-Google-DKIM-Signature field is another DKIM signature added by Gmail.

Conclusion

There’s obviously a lot of complexity lurking in those email headers. I’ve really learned a lot while researching this post and I hope that this information will be useful to you too.

Medusa: A Multi-Headed Tor Proxy

Andrew B. Collier — Mon, 04 Oct 2021 00:00:00 +0000

At Fathom Data we have a few projects which require us to send HTTP requests from an evolving selection of IP addresses. This post details one approach which uses Tor (The Onion Router) as a proxy.

What is a Proxy Server?

A proxy server acts as an intermediary between a client and a server. When a request goes through a proxy server there is no direct connection between the client and the server. The client connects to the proxy and the proxy then connects to the server. Requests and responses pass through the proxy.

Thanks to @axiematic for the figure.

HTTP & SOCKS Proxies

HTTP (HyperText Transfer Protocol) is the dominant protocol for information exchange on the internet. HTTP is connectionless. This means that the client (often a browser) sends a request to a server. The server then replies with a response. Once this interaction is over there is no persistent connection between the client and the server. Any further interactions require new connections.

An HTTP proxy uses the HTTP protocol for all interactions with the client and server. As a result, an HTTP proxy is only able to handle HTTP and HTTPS requests. An HTTP proxy is also able to filter or modify the content of the requests and responses passing through it.

SOCKS is another internet protocol. Whereas as HTTP is an application layer protocol (at the top of the OSI Model), SOCKS is a lower level protocol in the session layer.

A SOCKS proxy uses the SOCKS protocol. Since it’s secure (the name is an abbreviation for “SOCKet Secure”), a SOCKS proxy cannot understand the contents of requests or responses, so is unable to modify or filter them. Since it operates at a lower level in the networking hierarchy, a SOCKS proxy is also faster and more flexible than an HTTP proxy.

Tor Proxy Docker Image

We constructed a Docker image which uses the Tor network to expose both SOCKS and HTTP proxies. The image uses the following components:

Tor
HAProxy and
Privoxy.

The relationship between these components is detailed in the figure below.

Components in the Medusa Proxy (single headed configuration).

Tor

Tor provides an anonymous SOCKS proxy. The image will run multiple Tor instances, each of which will (in general) have a different exit node. This means that requests being routed through each instance will appear to come from a distinct IP address.

HAProxy

HAProxy is a high availability proxy server and load balancer (spreads requests across multiple services). HAProxy is used to distribute SOCKS requests across the Tor instances using a round robin scheduling strategy.

Privoxy

To provide for services that prefer to communicate via HTTP or cannot communicate via SOCKS, Privoxy is used to accept HTTP requests and forward them as SOCKS requests to HAProxy.

Running the Docker Image

Let’s spin up a container and take a look.

    docker run \
        -p 8800:8800 \
        -p 8888:8888 \
        -p 1080:1080 \
        -p 2090:2090 \
        datawookie/medusa-proxy

We’re mapped a lot of ports.

8800 — list of proxy URLs (as text/plain)
8888 — Privoxy port (HTTP protocol)
1080 — HAProxy port (SOCKS protocol) and
2090 — HAProxy statistics port.

Not all of them are required, but they all fulfill a distinct purpose. Below are some alternative ways to invoke the image.

# HTTP proxy on port 8888
docker run -p 8888:8888 datawookie/medusa-proxy
# SOCKS proxy on port 1080
docker run -p 1080:1080 datawookie/medusa-proxy
# Both HTTP and SOCKS proxies
docker run -p 8888:8888 -p 1080:1080 datawookie/medusa-proxy

Once we’ve got a running container we can set up a client to use the proxy. We’ll start with a browser and then look at curl on the command line.

Browser

Set up your browser to use the HTTP proxy. You could equally choose to use the SOCKS proxy, which will have an IP address of 127.0.0.1 and port 1080.

Once you’ve configured the proxy settings in your browser, head over to What Is My IP Address to check on your effective IP address. Refresh the page to confirm that the IP address changes.

Testing

Further testing is easier to do (and record) on the command line. To illustrate how the proxy works we’ll send requests to http://httpbin.org/ip to retrieve our effective IP address.

First let’s look at the set of IP addresses reported by the container. Below is an extract from the Docker logs.

2021-09-29 04:24:40 [INFO] Testing proxy (port 10000): 195.154.35.52.
2021-09-29 04:24:40 [INFO] Testing proxy (port 10001): 178.20.55.18.
2021-09-29 04:24:40 [INFO] Testing proxy (port 10002): 195.176.3.24.
2021-09-29 04:24:40 [INFO] Testing proxy (port 10003): 185.220.100.252.
2021-09-29 04:24:40 [INFO] Testing proxy (port 10004): 185.220.101.198.

Don’t worry about the port numbers (those are only relevant within the container). Note that there are five proxies (one for each Tor instance) and that each has a different IP address.

Now send out a series of requests.

curl http://httpbin.org/ip

{
  "origin": "195.154.35.52"
}

curl http://httpbin.org/ip

{
  "origin": "178.20.55.18"
}

curl http://httpbin.org/ip

{
  "origin": "195.176.3.24"
}

curl http://httpbin.org/ip

{
  "origin": "185.220.100.252"
}

curl http://httpbin.org/ip

{
  "origin": "185.220.101.198"
}

curl http://httpbin.org/ip

{
  "origin": "195.154.35.52"
}

Notice that each request appears to originate from a distinct IP address and that once we’ve cycled through all of the Tor instances, we wrap back to the first one.

Rotating

The exit nodes are periodically rotated. Some time later we see that we’re using a different set of IP addresses.

2021-09-29 04:39:13 [INFO] Testing proxy (port 10000): 171.25.193.20.
2021-09-29 04:39:13 [INFO] Testing proxy (port 10001): 199.249.230.87.
2021-09-29 04:39:14 [INFO] Testing proxy (port 10002): 185.220.101.132.
2021-09-29 04:39:14 [INFO] Testing proxy (port 10003): 199.249.230.184.
2021-09-29 04:39:15 [INFO] Testing proxy (port 10004): 23.129.64.161.

A Weakness

So this is great, but there’s one major weakness: if any one of the Tor instances became unavailable then the proxy would be marked as broken (regardless of whether the other Tor instances were fine or not). To get around this I have been running multiple containers, each assigned to a different port. With this setup, even if a few of the containers are marked as broken, there are still others which are considered healthy and able to accept requests. Although easy enough to automate, the logistics associated with this setup are a little onerous.

Wouldn’t it be convenient if there was just a single container which exposes multiple proxies, each of which is hooked up to a distinct set of Tor instances?

Beast with Many Heads

So, rather than exposing just a single unit per container, the Medusa proxy can cater for multiple proxy units or heads. What this means is that there can be multiple copies of the components illustrated in the diagram above. All heads are served from the same network location but on different ports.

There are a few environment variables which can be used to tweak the configuration:

TORS — Number of heads (default: 2)
TORS — Number of Tor instances (default: 5)
HAPROXY_LOGIN — Username for HAProxy (default: “admin”)
HAPROXY_PASSWORD — Password for HAProxy (default: “admin”)

Let’s give this a try. We’ll launch Medusa with 4 heads (each linking to 3 Tor instances) and only map the ports for the HTTP proxies.

    docker run \
        -e TORS=3 \
        -e HEADS=4 \
        -p 8800:8800 \
        -p 8888:8888 -p 8889:8889 -p 8890:8890 -p 8891:8891 \
        datawookie/medusa-proxy

So now we have four proxy connections at ports 8888, 8889, 8890 and 8891. If we look at the Docker logs then we see that there are 3 Tor endpoints for each head.

2021-09-30 08:05:34,606 [INFO] Testing proxies.
2021-09-30 08:05:34,606 [INFO] * Privoxy 0
2021-09-30 08:05:35,295 [INFO] Testing proxy (port 10000): 185.220.101.2.
2021-09-30 08:05:35,753 [INFO] Testing proxy (port 10001): 198.98.62.74.
2021-09-30 08:05:36,041 [INFO] Testing proxy (port 10002): 185.220.100.245.
2021-09-30 08:05:36,041 [INFO] * Privoxy 1
2021-09-30 08:05:36,652 [INFO] Testing proxy (port 10003): 37.123.163.58.
2021-09-30 08:05:37,304 [INFO] Testing proxy (port 10004): 37.187.196.70.
2021-09-30 08:05:37,634 [INFO] Testing proxy (port 10005): 77.68.20.217.
2021-09-30 08:05:37,634 [INFO] * Privoxy 2
2021-09-30 08:05:37,909 [INFO] Testing proxy (port 10006): 89.163.143.8.
2021-09-30 08:05:38,954 [INFO] Testing proxy (port 10007): 185.220.101.10.
2021-09-30 08:05:39,721 [INFO] Testing proxy (port 10008): 195.206.105.217.
2021-09-30 08:05:39,721 [INFO] * Privoxy 3
2021-09-30 08:05:40,050 [INFO] Testing proxy (port 10009): 185.220.100.243.
2021-09-30 08:05:41,010 [INFO] Testing proxy (port 10010): 185.220.101.43.
2021-09-30 08:05:41,415 [INFO] Testing proxy (port 10011): 185.185.170.27.

Proxy List

A proxy list is served as a plain text file on port 8800. This can be used to configure rotating proxies in clients.

Statistics

You can monitor the performance of the proxies via the statistics interface, which each HAProxy instance exposes via a port numbered sequentially from 2090.

Using Medusa with Scrapy

Technical Details

The image is derived from the official Alpine image, onto which Python 3, Tor, HAProxy and Privoxy are installed. The configuration files for each of the services are created from templates using Jinja templates.

{emayili} Right-to-Left

Andrew B. Collier — Fri, 24 Sep 2021 00:00:00 +0000

Yoav Raskin suggested that it would be useful to support right-to-left (RTL) documents in {emayili}, so that languages like Hebrew, Arabic and Aramaic would render properly. I’ll be honest, this was not something that I had previously considered. But agreed, it would be a cool feature.

library(emayili)

packageVersion("emayili")

[1] '0.5.5'

RTL in CSS

The first step to make this happen was writing a short CSS file, rtl.css.

.rtl {
  direction: rtl;
  color: green;
}

body {
  margin: 20px 0px;
}

figure {
  margin-bottom: 20px;
}

.rtl figcaption {
  direction: inherit;
}

This makes use of the CSS direction property (which by default is ltr), setting it to rtl. I’m also colouring the RTL text in green, just to make things super clear.

I put together a simple .Rmd file to test, rtl.Rmd, rendered it into a message and then dispatched.

envelope() %>%
  subject("Right-to-Left Text") %>%
  render("rtl.Rmd", css_files = "rtl.css")

The extra CSS is included via the css_files parameter.

🚨 If you want this to work nicely on Gmail then you’ll also need to set include_css = c("rmd", "highlight") because Gmail doesn’t currently appreciate Bootstrap CSS.

In the interests of brevity the contents of rtl.Rmd have been appended to the end of the post.

And this is what the delivered email looks like:

Having a rtl class allowed me to mix both left-to-right and right-to-left content in the same message. In reality this is a rather unlikely requirement. To use RTL for the entire message just adapt the CSS as follows:

body {
  direction: rtl;
}

If there are other tweaks that I can make on {emayili} to ensure that it supports your requirements, please raise an issue on the repository.

The {emayili} package is developed & supported by Fathom Data.

Here are the contents of rtl.Rmd:

--------
output: html_document
--------

{r setup, include=FALSE}
knitr::opts_chunk$set(echo = TRUE)


Lorem ipsum dolor sit amet, consectetur adipiscing elit. Integer non magna
magna. Etiam eu sapien nulla. Cras ac lectus non urna scelerisque porttitor ac
non tortor.

<div class="rtl">
צחוק המעבורת, מהדובים של הכרית החופשית. מי רוצה לפספס, עצוב בחצר האחורית, כד
רך. עכשיו הכאב של הכאב הוא לא הכאב של הטרדה, הצורך של אגמי החלבון של התפוז.
אלמנט כלשהו, וגם לא יסוד כימי של נדל"ן, מראית עין של מגוון כלי רכב לנחות, רגע
לפני הספד החיים. אניאס הוא מחבר רכיב המחיר. אין פחד אלא אם להיות כנים. אני חי
במייסר העצוב של הכדורגל שלי, בסמך הזן כמובן. אבל המענה והקשתות, או קצות האורקים.
אל האגם או לפניו. הזמן של העמק לוקח. אפילו הפחד מהפחד, הקשתות או המעבורת, אתת
האינטרנט.
</div>

{r pressure, out.width="50%", fig.cap="כיתוב דמות", fig.class="rtl"}
par(mar = c(5, 4, 1, 1))
plot(pressure, xlab = "טֶמפֶּרָטוּרָה", ylab = "לַחַץ", mar = c(5, 4, 0, 2))


<div class="rtl">
המחלה נושאת את הסבירות לסוף השבוע, והפחד מהכדורגל קשה. אבל האריה והאריה והילדים
נמצאים במיטה. זה ברוטב צריך להגיד. אבל מבחינתו, צרות החיים מותירות הרבה חיים,
הקריקטורה זקוקה לארוסים. עכשיו אני שונא את הגורמים, לא את הכאב ולא את הסביבה,
הכניסה קלה. עכשיו הכניסה לחיי.
</div>

Cras fringilla nunc in tellus sagittis accumsan. Mauris nisi tellus, congue sit
amet turpis nec, accumsan lacinia neque. Nulla facilisi.

{emayili} Styling Figures

Andrew B. Collier — Thu, 23 Sep 2021 00:00:00 +0000

By default <img> tags are wrapped in a tight <p></p> embrace by {knitr}. In general this works really well. However, I want to have more control over image formatting for {emayili}.

Adding a Hook

I’d like to have the <img> tags wrapped by <figure></figure>. It’d also be useful to have the option of inserting a <figcaption>. To support these requirements I added a {knitr} hook which adds in these tags. It responds to the following chunk parameters (all optional):

out.width — the figure width
fig.cap — figure caption
fig.alt — figure alternative text and
fig.class — class attached to the <figure> tag.

Example

First we’ll try out the figure caption with the following .Rmd file.

--------
output: html_document
--------

{r fig.cap="Mercury Vapour"}
plot(pressure)

Rendering this file, attaching to an envelope object and sending it with {emayili} yields the message below.

Nothing too exciting, just a caption neatly displayed below the plot.

Next let’s try attaching a class to the <figure> tag. We’ll use the CSS file figures.css. This file needs to be specified with the css_files argument to render().

body {
  margin: 30px 0px;
}

figure {
  margin-bottom: 20px;
}

figure.center {
  text-align: center;
}

figcaption {
  color: white;
  text-align: left;
  background-color: #9fa4a7;
  padding: 10px 20px;
}

figcaption::before {
  content: "Figure: ";
}

In the .Rmd file we use the fig.class parameter to apply the center class to the <figure>.

--------
output: html_document
--------

{r out.width="75%", fig.class="center"}
plot(pressure)

And this is what the rendered message looks like.

The <img> is now centered in the <figure>.

Finally we’ll bring it all together by styling a figure caption.

--------
output: html_document
--------

{r out.width="75%", fig.class="center", fig.cap="Mercury Vapour"}
plot(pressure)

And the result in your inbox.

Since many automated reports use figures, the ability to style those figures should be very useful.

The {emayili} package is developed & supported by Fathom Data.

{emayili} Managing CSS

Andrew B. Collier — Wed, 22 Sep 2021 00:00:00 +0000

I love the clean simplicity of an R Markdown document. But sometimes it can feel a little bare and utilitarian. This is especially the case if it’s rendered into the body of an email. How about injecting a little more pizzazz?

library(emayili)

msg <- envelope()

(In|Ex)cluding Rendered CSS

By default the knitting process for an R Markdown document will inject CSS from Bootstrap and highlightjs. When you render() an R Markdown document in {emayili} these CSS will be included in the <head> of the HTML document. This actually results in quite a lot of content, often more than contained in the <body> of the document.

If you want to generate a more lightweight message then you can use include_css option to turn off the CSS.

msg %>%
  # Do include rendered CSS (default).
  render("styling.Rmd", include_css = TRUE)

msg %>%
  # Don't include rendered CSS.
  render("styling.Rmd", include_css = FALSE)

Let’s take a look at those two options. This is what it looks like in Thunderbird when the rendered CSS is included.

And this is it without the rendered CSS.

Custom CSS

Now, what about including some custom CSS? Use the css_files argument to specify one or more CSS files that will get baked into the message.

msg %>%
  render("styling.Rmd", include_css = FALSE, css_files = "styling.css")

Here’s the contents of styling.css:

@import url('https://fonts.googleapis.com/css2?family=Pacifico&display=swap');
@import url('https://fonts.googleapis.com/css2?family=Roboto&display=swap');

h2 {
  font-family: 'Pacifico', cursive;
}

p {
  font-family: 'Roboto', sans-serif;
}

body {
  background-color: grey;
  color: white;
  margin: 0 30px;
}

hr {
  height: 1px;
  background-color: white;
  border: none;
}

We’re pulling in some custom Google Fonts, changing the background colour of the page and adding in some margins, along with a few other tweaks.

And this is what it looks like. I can’t claim to have any aesthetic sense at all. Sorry. I’m sure that you can do a lot better!

With these options at your disposal there’s really no excuse for creating “ordinary” emails. Now’s your chance to be extra-ordinary.

{emayili} R Markdown Parameters

Andrew B. Collier — Mon, 20 Sep 2021 00:00:00 +0000

I don’t frequently use parameters in R Markdown documents. So the initial implementation of render() in {emayili} did not cater for them. A small tweak makes it possible though.

You can install the update from GitHub as follows.

remotes::install_github("datawookie/emayili", ref = "v0.5.2")

Then load the package.

library(emayili)

# A couple of options to display message body.
#
options(
  envelope.details = TRUE,
  envelope.invisible = FALSE
)

Create an empty message object.

msg <- envelope()

Suppose you have the following simple R Markdown document, params.Rmd, which you want to render into the body of an email.

--------
title: "Message with Parameter"
output: html_document
params:
  value: 13
--------

The value is `r params$value`.

We’ll start by rendering it using the default parameter value.

msg %>%
  subject("Default parameter value") %>%
  render("params.Rmd", include_css = FALSE)

Date: Mon, 20 Sep 2021 08:20:59 GMT
Subject: Default parameter value
X-Mailer: {emayili}-0.5.2
MIME-Version: 1.0
Content-Type: multipart/related; boundary="9554b959"

--9554b959
Content-Type: text/html; charset=utf-8
Content-Disposition: inline

<html>
<head></head>
<body>
<div class="container-fluid main-container">
<div id="header">
<h1 class="title toc-ignore">Message with Parameter</h1>
</div>
<p>The value is 13.</p>
</div>
</body>
</html>

--9554b959--

As expected, the default value for the parameter comes through.

Now let’s test a different value.

msg %>%
  subject("Better parameter value") %>%
  render("params.Rmd", params = list(value = 42), include_css = FALSE)

Date: Mon, 20 Sep 2021 08:20:59 GMT
Subject: Better parameter value
X-Mailer: {emayili}-0.5.2
MIME-Version: 1.0
Content-Type: multipart/related; boundary="a099637d"

--a099637d
Content-Type: text/html; charset=utf-8
Content-Disposition: inline

<html>
<head></head>
<body>
<div class="container-fluid main-container">
<div id="header">
<h1 class="title toc-ignore">Message with Parameter</h1>
</div>
<p>The value is 42.</p>
</div>
</body>
</html>

--a099637d--

🚀 Success: the specified parameter value now appears. If you use parameters in your R Makrkdown documents, then you can be confident that these will now work in {emayili} too.

{emayili} Rendering R Markdown

Andrew B. Collier — Fri, 17 Sep 2021 00:00:00 +0000

In a previous post I documented a new feature in {emayili}, the ability to render Plain Markdown directly into the body of an email message.

Today I’m announcing the release of a new minor version, 0.5.0, in which {emayili} is now able to render R Markdown into an email. This is a major leap forward for the package.

I should mention that this capability (rendering R Markdown into an email) is already available in the {mailmerge} package, but that only works with Gmail.

Install & Load

You can install this version directly from GitHub as follows.

remotes::install_github("datawookie/emayili", ref = "v0.5.0")

This version was also published on CRAN on 17 September 2021.

Now load the package.

library(emayili)

# A couple of options to display message body.
#
options(
  envelope_details = TRUE,
  envelope_invisible = FALSE
)

Check on the installed version.

packageVersion("emayili")

[1] '0.5.0'

A Simple R Markdown File

For the purposes of illustration let’s use a super simple R Markdown file, pi.Rmd.

--------
title: "Approximating Pi"
output: html_document
--------

{r}
22 / 7

Create an empty message object.

msg <- envelope()

And then render pi.Rmd into the body of the message. In this case I’m going to set the include_css option to FALSE so that the copious volume of CSS is suppressed in the resulting HTML.

msg %>% render("pi.Rmd", include_css = FALSE)

Date: Fri, 17 Sep 2021 10:16:09 GMT
X-Mailer: {emayili}-0.5.0
MIME-Version: 1.0
Content-Type: multipart/related; boundary="b9173ea5"

--b9173ea5
Content-Type: text/html; charset=utf-8
Content-Disposition: inline

<html><body>
<div class="container-fluid main-container">
<div id="header">
<h1 class="title toc-ignore">Approximating Pi</h1>
</div>
<pre class="r"><code>22 / 7</code></pre>
<pre><code>[1] 3.142857</code></pre>
</div>
</body></html>

--b9173ea5--

The document is inserted as a separate MIME chunk with type text/html.

Let’s move onto a more interesting example.

R Markdown File from Template

Create an R Markdown file using the "github_document" template.

rmd <- "gh-doc.Rmd"

rmarkdown::draft(
  rmd,
  template = "github_document",
  package = "rmarkdown",
  edit = FALSE
)

--------
title: "Untitled"
output: github_document
--------

{r setup, include=FALSE}
knitr::opts_chunk$set(echo = TRUE)


## GitHub Documents

This is an R Markdown format used for publishing markdown documents to GitHub. 
When you click the **Knit** button all R code chunks are run and a markdown 
file (.md) suitable for publishing to GitHub is generated.

## Including Code

You can include R code in the document as follows:

{r cars}
summary(cars)


## Including Plots

You can also embed plots, for example:

{r pressure, echo=FALSE}
plot(pressure)


Note that the `echo = FALSE` parameter was added to the code chunk to prevent 
printing of the R code that generated the plot.

Now we’ll add some addresses to the message object and render the R Markdown.

msg %>%
  to("bob@google.com") %>%
  from("alice@google.com") %>%
  subject("Rendering an R Markdown Document") %>%
  render(rmd)

Dispatching the email results in the message below.

The message doesn’t need to look this generic. You can make your R Markdown documents as fancy as you want and {emayili} will be happy to send them.

Conclusion

If you haven’t tried {emayili} before, now is a great time to give it a shot. And if you use R for automated reporting, then being able to render R Markdown directly into an email should simplify your workflow.

I gave a talk about these new features (and most of the old features!) at the Birmingham R User Group. The slides are available here.

{emayili} Rendering Plain Markdown

Andrew B. Collier — Fri, 10 Sep 2021 00:00:00 +0000

We’ve been able to attach text and HTML content to messages with {emayili}. But something that I’ve really been wanting to do is render markdown directly into an email.

In version 0.4.19 I’ve added the ability to directly render plain markdown into a message.

library(emayili)

The render() method will handle plain markdown either as a character vector or from a file.

Markdown String

Let’s start with markdown in a character vector.

email <- envelope() %>%
  render("[This](https://www.google.com) is a link.")

What’s the raw email document look like?

print(email, TRUE)

Date: Fri, 10 Sep 2021 03:42:47 GMT
X-Mailer: {emayili}-0.4.19
MIME-Version: 1.0
Content-type: multipart/mixed; boundary="e28a25a2a39253440353e12c623"

--e28a25a2a39253440353e12c623
Content-Type: text/html; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

<p><a href=3D"https://www.google.com">This</a> is a link.</p>

--e28a25a2a39253440353e12c623--

The markdown has been translated into HTML and inserted as a text/html MIME element.

Markdown in a File

You’re more likely to have your markdown in a file. Suppose that the file message.md contains the following:

[This](https://www.google.com) is a link.

- One
- Two
- Three

![](https://cran.r-project.org/Rlogo.svg)

The render() function will identify its argument as a file path and render the contents of the file.

email <- envelope() %>%
  render("message.md")

And this is what the resulting MIME message looks like:

print(email, TRUE)

Date: Fri, 10 Sep 2021 03:42:47 GMT
X-Mailer: {emayili}-0.4.19
MIME-Version: 1.0
Content-type: multipart/mixed; boundary="8301b826a4415341f24323d192f"

--8301b826a4415341f24323d192f
Content-Type: text/html; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

<p><a href=3D"https://www.google.com">This</a> is a link.</p>
<ul>
<li>One</li>
<li>Two</li>
<li>Three</li>
</ul>
<p><img src=3D"https://cran.r-project.org/Rlogo.svg" alt=3D"" /></p>

--8301b826a4415341f24323d192f--

This new feature partially scratches a persistent itch. But I’ll only be satisfied when I can render R markdown straight into an email. Stay tuned.

{clockify} Time Tracking from R

Andrew B. Collier — Thu, 09 Sep 2021 00:00:00 +0000

At Fathom Data we use Clockify to keep detailed records of the time that we spend working on our clients’ projects. Up until fairly recently we manually generated timesheets at the end of each month that were sent through to the clients along with their invoices. Our experience has been that providing detailed timesheets helps foster trust and transparency. However, with a growing team and an expanding clientele, generating these timesheets has become progressively more laborious. Time to automate!

Luckily Clockify exposes an extensive API. I built a small R package, {clockify} which is a wrapper around the Clockify API, making it easy to tap into our timesheet data. We’re then using R Markdown to generate some pretty sweet automated timesheets and reports. This reporting all happens automatically courtesy of GitLab CI. But that’s a story for another day. Let’s take a look at {clockify}.

This post documents {clockify}-0.0.9.

Installation

You can install the latest development version straight from GitHub.

remotes::install_github("datawookie/clockify")

Or you can install a stable version from CRAN.

install.packages("clockify")

The package documentation is available here.

Once you’ve installed the package, load it into R and check the version.

library(clockify)

packageVersion("clockify")

[1] '0.0.9'

API Key

You’re going to need to have an API key from your Clockify account. If you don’t yet have an account, create one. Then retrieve the API key from the account settings.

Get Started

Set the API key to be used for the session. I store my API key in an environment variable called CLOCKIFY_API_KEY.

CLOCKIFY_API_KEY = Sys.getenv("CLOCKIFY_API_KEY")

set_api_key(CLOCKIFY_API_KEY)

Let’s turn on logging so we can see what’s happening behind the scenes.

library(logger)

log_threshold(DEBUG)

Workspaces

Clockify uses workspaces to group people and projects. Every Clockify account is automatically associated with a default workspace, but may be linked to other workspaces too.

Use workspaces() to retrieve a list of available workspaces.

workspaces()

2021-09-09 08:21:32 — GET /workspaces

# A tibble: 2 × 2
  workspace_id name       
  <chr> <chr>      
1 5ef46294df73063139f60bfc Fathom Data
2 61343c45ab05e02be2c8c1fd Personal

I have two workspaces associated with my account, Personal and Work.

Select the Personal workspace. All subsequent operations will apply within this workspace.

workspace("61343c45ab05e02be2c8c1fd")

2021-09-09 08:21:32 — Set active workspace -> 61343c45ab05e02be2c8c1fd.

[1] "61343c45ab05e02be2c8c1fd"

Users

The users() function generates a table of all users linked to the workspace.

users()

2021-09-09 08:21:32 — GET /workspaces/61343c45ab05e02be2c8c1fd/users

# A tibble: 2 × 3
  user_id user_name status
  <chr> <chr> <chr> 
1 5f227e0cd7176a0e6e754409 Andrew ACTIVE
2 5ef46293df73063139f60bf5 Emma ACTIVE

Retrieve information from your user profile.

user()

2021-09-09 08:21:32 — GET /user

# A tibble: 1 × 3
  user_id user_name status
  <chr> <chr> <chr> 
1 5f227e0cd7176a0e6e754409 Andrew ACTIVE

Clients

So, who do you work for? Let’s get a list of clients associated with the workspace.

clients()

2021-09-09 08:21:33 — GET /workspaces/61343c45ab05e02be2c8c1fd/clients

# A tibble: 2 × 3
  client_id workspace_id client_name
  <chr> <chr> <chr>      
1 61343c6c00dc8f48962b9be9 61343c45ab05e02be2c8c1fd Community  
2 61343c5d00dc8f48962b9be3 61343c45ab05e02be2c8c1fd Fathom Data

Projects

Get a table of projects. The endpoint used to retrieve projects is paginated, so there are multiple API calls to retrieve the complete list.

projects()

2021-09-09 08:21:33 — GET /workspaces/61343c45ab05e02be2c8c1fd/projects
2021-09-09 08:21:33 — Page contains 2 results.
2021-09-09 08:21:33 — GET /workspaces/61343c45ab05e02be2c8c1fd/projects
2021-09-09 08:21:33 — Page is empty.
2021-09-09 08:21:33 — API returned 2 results.

# A tibble: 2 × 4
  project_id project_name client_id billable
  <chr> <chr> <chr> <lgl>   
1 6134506c777d5361dcdeb3b5 {cex} 61343c5d00dc8f48962b9be3 TRUE    
2 61343c9ba15c1d53ad33369f {clockify} 61343c5d00dc8f48962b9be3 FALSE

I’m logging time on two projects, {clockify} and {emayili}.

Let’s change the logging threshold, because it’s going to get noisy.

log_threshold(INFO)

Time Entries

The really interesting data is captured in the time entries: how much time am I spending on each of the projects and what am I doing?

Retrieve Time Entries

Retrieve the time entries for the authenticated user.

time_entries()

# A tibble: 8 × 4
  id project_id description duration
  <chr> <chr> <chr> <dbl>
1 61343cc1777d5361dcdea70a 61343c9ba15c1d53ad33369f Set up GitHub Acti… 12  
2 61343d06777d5361dcdea729 61343c9ba15c1d53ad33369f Make coffee 5  
3 61343d27ab05e02be2c8c266 61343c9ba15c1d53ad33369f Populate README.Rmd 68  
4 613448d7777d5361dcdead37 61343c9ba15c1d53ad33369f Add GET /workspace… 12.0
5 61344bcad01d3b4a27a82310 61343c9ba15c1d53ad33369f Add GET /workspace… 31.8
6 6134548f00dc8f48962bace7 6134506c777d5361dcdeb3b5 Add GET /ticker/{s… 16.0
7 6134585a777d5361dcdebc5c 6134506c777d5361dcdeb3b5 Add GET /tickers/{… 16.5
8 61345c45d01d3b4a27a833c7 6134506c777d5361dcdeb3b5 Add GET /last_pric… 31.6

You can also get the start and end times (as well as a bunch of other fields) for each entry by setting concise = FALSE.

time_entries(concise = FALSE) %>% select(id, starts_with("time"))

# A tibble: 8 × 3
  id time_start time_end           
  <chr> <dttm> <dttm>             
1 61343cc1777d5361dcdea70a 2021-09-03 05:15:00 2021-09-03 05:27:00
2 61343d06777d5361dcdea729 2021-09-03 05:27:00 2021-09-03 05:32:00
3 61343d27ab05e02be2c8c266 2021-09-03 05:45:00 2021-09-03 06:53:00
4 613448d7777d5361dcdead37 2021-09-05 04:34:31 2021-09-05 04:46:30
5 61344bcad01d3b4a27a82310 2021-09-05 04:47:06 2021-09-05 05:18:56
6 6134548f00dc8f48962bace7 2021-09-05 05:24:30 2021-09-05 05:40:29
7 6134585a777d5361dcdebc5c 2021-09-05 05:40:42 2021-09-05 05:57:13
8 61345c45d01d3b4a27a833c7 2021-09-05 05:57:25 2021-09-05 06:29:01

Retrieve time entries for another user specified by their user ID.

time_entries(user_id = "5ef46293df73063139f60bf5")

# A tibble: 2 × 4
  id project_id description duration
  <chr> <chr> <chr> <dbl>
1 613630ebba4b374e57155a72 61343c9ba15c1d53ad33369f Another test entry 87  
2 613630cb89516b1767a56a08 61343c9ba15c1d53ad33369f Creating hex logo 45.4

Insert Time Entry

What about inserting a new time entry? No problem, call time_entry_insert() and supply the details.

prepare_cran_id <- time_entry_insert(
 project_id = "61343c9ba15c1d53ad33369f",
 start = "2021-08-30 08:00:00",
 end = "2021-08-30 10:30:00",
 description = "Prepare for CRAN submission"
)

The ID for the newly inserted time entry is returned.

prepare_cran_id

[1] "6139c40ff1d18b7ed667a7bd"

Confirm that it has been inserted.

time_entries(concise = FALSE) %>% select(id, description, duration)

# A tibble: 9 × 3
  id description duration
  <chr> <chr> <dbl>
1 6139c40ff1d18b7ed667a7bd Prepare for CRAN submission 150  
2 61343cc1777d5361dcdea70a Set up GitHub Actions 12  
3 61343d06777d5361dcdea729 Make coffee 5  
4 61343d27ab05e02be2c8c266 Populate README.Rmd 68  
5 613448d7777d5361dcdead37 Add GET /workspaces/{workspaceId}/projects/… 12.0
6 61344bcad01d3b4a27a82310 Add GET /workspaces/{workspaceId}/projects/… 31.8
7 6134548f00dc8f48962bace7 Add GET /ticker/{symbol1}/{symbol2} 16.0
8 6134585a777d5361dcdebc5c Add GET /tickers/{symbol1}/{symbol2}/.../{s… 16.5
9 61345c45d01d3b4a27a833c7 Add GET /last_price/{symbol1}/{symbol2} 31.6

Delete Time Entry

What about deleting a time entry? Easy, just call time_entry_delete() and supply the time entry ID.

time_entry_delete(prepare_cran_id)

[1] TRUE

Confirm that it has been deleted.

time_entries(concise = FALSE) %>% select(id, description, duration)

# A tibble: 8 × 3
  id description duration
  <chr> <chr> <dbl>
1 61343cc1777d5361dcdea70a Set up GitHub Actions 12  
2 61343d06777d5361dcdea729 Make coffee 5  
3 61343d27ab05e02be2c8c266 Populate README.Rmd 68  
4 613448d7777d5361dcdead37 Add GET /workspaces/{workspaceId}/projects/… 12.0
5 61344bcad01d3b4a27a82310 Add GET /workspaces/{workspaceId}/projects/… 31.8
6 6134548f00dc8f48962bace7 Add GET /ticker/{symbol1}/{symbol2} 16.0
7 6134585a777d5361dcdebc5c Add GET /tickers/{symbol1}/{symbol2}/.../{s… 16.5
8 61345c45d01d3b4a27a833c7 Add GET /last_price/{symbol1}/{symbol2} 31.6

Test Coverage

I’ve started writing unit tests. You can check on the current test coverage here. Evidently there’s still work to be done!

Conclusion

Exposing an API adds a lot of value to Clockify as a time tracking product. If you’re using Clockify, then you’ll find the {clockify} package convenient and useful for accessing those data. It plays particularly well in automated reporting scenarios.

Setting up a Tiny HTTP Proxy

Andrew B. Collier — Wed, 08 Sep 2021 00:00:00 +0000

It’s often handy to have access to a HTTP proxy. I use this recipe from time to time to quickly fling together a proxy server which I can use to relay HTTP requests from a different origin.

EC2 Instance

Create an EC2 instance in the region you want the proxy to reside. You can go with the smallest instance available. I’m using a t3.nano.

Install an Ubuntu AMI.

If you are planning on keeping the proxy running indefinitely, then it will be a good idea to associate an Elastic IP address with this instance.

Install Tinyproxy

Connect to the EC2 instance via SSH and install Tinyproxy.

sudo apt-get update
sudo apt-get install tinyproxy

Restrict Access

Now you probably don’t want the proxy to be accessible to everybody, so lock it down to a specific selection of IP addresses.

Find the IP address of the machine that will have access to the proxy. You can either visit https://whatismyipaddress.com/ or run the following shell command.

curl ipecho.net/plain

3.91.59.140

Add a security group which allows access from this IP on port 8888.

You could be a lot more permissive and allow access on port 8888 from all IP addresses. But this might end in tears.

Configure Tinyproxy

Edit the configuration for Tinyproxy at /etc/tinyproxy/tinyproxy.conf.

Allowing Access from Nominated IP Addresses

Find the Allow section in the configuration file and add a line for the machine which will be accessing the proxy. Specify the IP address that you found earlier.

Allow 3.91.59.140

This setting should be consistent with whatever access you have permitted in the security group.

An alternative (and possibly more flexible) approach is to not define any Allow rules but handle access via security groups.

If you want to live dangerously then you can allow access from anywhere.

Allow 0.0.0.0/0

Be warned though, this too may end in tears.

Basic Authentication

What about ensuring that only authenticated users have access to the service? No problem, just add a BasicAuth entry to the configuration file.

BasicAuth alice izlGVukLF8bSQuuzZKg

Restart

After the configuration has been adjusted, restart Tinyproxy.

sudo service tinyproxy restart

Test

Now test the proxy. Suppose, for example, that the proxy is running on a machine with IP address 3.11.245.24.

# Access HTTP(S) site via proxy (assuming no authentication required).
curl --proxy 3.11.245.24:8888 http://ipecho.net/plain
curl --proxy 3.11.245.24:8888 https://ipecho.net/plain

# Provide credentials for basic authentication.
curl --proxy alice:izlGVukLF8bSQuuzZKg@3.11.245.24:8888 http://ipecho.net/plain

In each case you should get back the IP address of the proxy server.

Once you’ve confirmed that it works, add it to the system or browser settings. You’re sorted! 🚀

{emayili} Interpolating Message Content

Andrew B. Collier — Fri, 03 Sep 2021 00:00:00 +0000

A small new feature added to {emayili}: the ability to interpolate content into the email message body.

Load Package

Load the {emayili} package and create a message skeleton.

library(emayili)

options(
  # Always print message body.
  envelope_details = TRUE,
  # Print message from pipeline.
  envelope_invisible = FALSE
)

# Create a message skeleton.
#
email <- envelope()

Body

Now let’s look at different ways to add a text body.

If you know precisely what your message is going to say, then you can just write it straight into the message body.

email %>% text("Hello Alice!")

Date: Fri, 03 Sep 2021 09:07:10 GMT
X-Mailer: {emayili}-0.4.17
MIME-Version: 1.0
Content-type: multipart/mixed; boundary="1133f2d1f303af212e1f293e2c361c"

--1133f2d1f303af212e1f293e2c361c
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: 7bit

Hello Alice!

--1133f2d1f303af212e1f293e2c361c--

But what if a part of the message is stored in a variable? Use the {glue} syntax for variable interpolation.

name <- "Bob"

email %>% text("Hello {name}!")

Date: Fri, 03 Sep 2021 09:07:10 GMT
X-Mailer: {emayili}-0.4.17
MIME-Version: 1.0
Content-type: multipart/mixed; boundary="1133f2d1f303af212e1f293e2c361c"

--1133f2d1f303af212e1f293e2c361c
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: 7bit

Hello Bob!

--1133f2d1f303af212e1f293e2c361c--

This should make generating automated (yet personalised!) emails a lot simpler. For example, here’s how you’d go about creating emails from a data frame with columns holding names and email addresses.

tribble(
  ~email, ~name,
  "alice@google.com", "Alice",
  "bob@yahoo.com", "Bob"
) %>%
  pwalk(function(email, name) {
    email <- envelope(to = email) %>% text("Hello {name}!")
    print(email)
  })

Date: Fri, 03 Sep 2021 09:07:10 GMT
To: alice@google.com
X-Mailer: {emayili}-0.4.17
MIME-Version: 1.0
Content-type: multipart/mixed; boundary="153ce203c1ff10232f22237eb5"

--153ce203c1ff10232f22237eb5
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: 7bit

Hello Alice!

--153ce203c1ff10232f22237eb5--
Date: Fri, 03 Sep 2021 09:07:10 GMT
To: bob@yahoo.com
X-Mailer: {emayili}-0.4.17
MIME-Version: 1.0
Content-type: multipart/mixed; boundary="92832121424fb5133d2a1220371b"

--92832121424fb5133d2a1220371b
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: 7bit

Hello Bob!

--92832121424fb5133d2a1220371b--

Unleash your inner spam king.

DEV Community: Andrew B. Collier

HCRIS Field Labels

Resources

Fathoming Email Headers

Typical Headers

Original Message Content

Number of Headers

Source Header Fields

Date

From and To

Subject

MIME-Version

Content-Type

Content-Disposition

wkszanyzfl .gt_table {

wkszanyzfl .gt_heading {

wkszanyzfl .gt_title {

wkszanyzfl .gt_subtitle {

wkszanyzfl .gt_bottom_border {

wkszanyzfl .gt_col_headings {

wkszanyzfl .gt_col_heading {

wkszanyzfl .gt_column_spanner_outer {

wkszanyzfl .gt_column_spanner_outer:first-child {

wkszanyzfl .gt_column_spanner_outer:last-child {

wkszanyzfl .gt_column_spanner {

wkszanyzfl .gt_group_heading {

wkszanyzfl .gt_empty_group_heading {

wkszanyzfl .gt_from_md > :first-child {

wkszanyzfl .gt_from_md > :last-child {

wkszanyzfl .gt_row {

wkszanyzfl .gt_stub {

wkszanyzfl .gt_summary_row {

wkszanyzfl .gt_first_summary_row {

wkszanyzfl .gt_grand_summary_row {

wkszanyzfl .gt_first_grand_summary_row {

wkszanyzfl .gt_striped {

wkszanyzfl .gt_table_body {

wkszanyzfl .gt_footnotes {

wkszanyzfl .gt_footnote {

wkszanyzfl .gt_sourcenotes {

wkszanyzfl .gt_sourcenote {

wkszanyzfl .gt_left {

wkszanyzfl .gt_center {

wkszanyzfl .gt_right {

wkszanyzfl .gt_font_normal {

wkszanyzfl .gt_font_bold {

wkszanyzfl .gt_font_italic {

wkszanyzfl .gt_super {

wkszanyzfl .gt_footnote_marks {

Content-Transfer-Encoding

Content-MD5

Transit Header Fields

Message-ID

Received

qiolitmyfl .gt_table {

qiolitmyfl .gt_heading {

qiolitmyfl .gt_title {

qiolitmyfl .gt_subtitle {

qiolitmyfl .gt_bottom_border {

qiolitmyfl .gt_col_headings {

qiolitmyfl .gt_col_heading {

qiolitmyfl .gt_column_spanner_outer {

qiolitmyfl .gt_column_spanner_outer:first-child {

qiolitmyfl .gt_column_spanner_outer:last-child {

qiolitmyfl .gt_column_spanner {

qiolitmyfl .gt_group_heading {

qiolitmyfl .gt_empty_group_heading {

qiolitmyfl .gt_from_md > :first-child {

qiolitmyfl .gt_from_md > :last-child {

qiolitmyfl .gt_row {

qiolitmyfl .gt_stub {

qiolitmyfl .gt_summary_row {

qiolitmyfl .gt_first_summary_row {

qiolitmyfl .gt_grand_summary_row {

qiolitmyfl .gt_first_grand_summary_row {

qiolitmyfl .gt_striped {

qiolitmyfl .gt_table_body {

qiolitmyfl .gt_footnotes {

qiolitmyfl .gt_footnote {

qiolitmyfl .gt_sourcenotes {

`Date`

`From` and `To`

`Subject`

`MIME-Version`

`Content-Type`

`Content-Disposition`

`Content-Transfer-Encoding`

`Content-MD5`

`Message-ID`

`Received`

`X-Originating-Ip`

`Delivered-To` and `Return-Path`

`Authentication-Results`

`DKIM-Signature`