DEV Community: Alex Sharp 🛠sharesecret.co

JSON Parsing with Go Embedded Structs

Alex Sharp 🛠sharesecret.co — Thu, 11 Jun 2020 21:48:42 +0000

Go has a feature called embedding that allows you to create higher level interfaces by combining existing interfaces. But embedding can also be applied to structs, which is very useful for JSON parsing/decoding.

Consider the common case of decoding JSON objects that have a timestamp field, in this case encoded as a unix integer timestamp, as JSON has no native date support (this same solution could be applied to a iso-8601 encoded date string).

Here's a PriceTick json object and struct that will decode but isn't quite what we want. Ideally we want Timestamp to be a time.Time object rather than an int64.

{
    "timestamp": 1591745820,
    "price": 321.655,
    "open": 321.68,
    "high": 321.7,
    "low": 321.61,
    "close": 321.61,
    "volume": 2608,
    "vwap": 321.66511
}

type PriceTick struct {
    Timestamp int64     `json:"timestamp"`
    Price     float64   `json:"price"`
    Volume    float64   `json:"volume"`
    Open      float64   `json:"open"`
    High      float64   `json:"high"`
    Low       float64   `json:"low"`
    Close     float64   `json:"close"`
    VWAP      float64   `json:"vwap"`
}

Go lets you do custom json decoding on an object by defining the UnmarshalJSON interface method on a struct. But we don't want to do it for the entire PriceTick object, just the timestamp property.

We can define a custom Timestamp struct to hold the time object, but then you have to dig into the object whenever you want to access the time.Time object it's holding, which is not ideal if we want to replicate the native object structure we're reading from JSON.

This is where embedded structs shine.

We can modify the definition above to point to a struct that effectively wraps a time.Time object, and only do custom json decoding on that object.

// Timestamp wraps a time object encoded as a int64 unix timestamp
// This can be used in structs and it automatically handles decoding
// int64 unix timestamps into a time.Time object.
type Timestamp struct {
    time.Time
}

// PriceTick is a tick returned from the TimeAndSales endpoint
type PriceTick struct {
    Timestamp Timestamp `json:"timestamp"`
    Time      string    `json:"time"`
    Price     float64   `json:"price"`
    Volume    float64   `json:"volume"`
    Open      float64   `json:"open"`
    High      float64   `json:"high"`
    Low       float64   `json:"low"`
    Close     float64   `json:"close"`
    VWAP      float64   `json:"vwap"`
}

// UnmarshalJSON decodes an int64 timestamp into a time.Time object
func (p *Timestamp) UnmarshalJSON(bytes []byte) error {
    // 1. Decode the bytes into an int64
    var raw int64
    err := json.Unmarshal(bytes, &raw)

    if err != nil {
        fmt.Printf("error decoding timestamp: %s\n", err)
        return err
    }

    // 2 - Parse the unix timestamp
    *&p.Time = time.Unix(raw, 0)
    return nil
}

By embedding the time.Time property in the Timestamp struct without a property name, we can call any time.Time methods directly on a Timestamp object.

From the Go documentation on embedding:

When we embed a type, the methods of that type become methods of the outer type, but when they are invoked the receiver of the method is the inner type, not the outer one.

A Disabled React Component

Alex Sharp 🛠sharesecret.co — Thu, 07 May 2020 09:04:37 +0000

Fairly often in a React app you need to disable a component based on a prop or some state. Using a Disabled component is a clean and reusable way to handle that conditional logic.

You could choose to implement disabled by not showing the component at all (eg setting display: none), or you can do as I've done here and change the opacity to gray out the component. Most importantly, you can easily disable click events by setting the pointerEvents css property to none.

Automatically Assign Elastic IPs to Elastic Beanstalk Instances

Alex Sharp 🛠sharesecret.co — Tue, 18 Feb 2020 16:48:17 +0000

I've had a need for a while to automatically assign elastic IP addresses to newly launched Elastic Beanstalk instances. This is super useful if you're using an external database host like Compose, and you only want to allow connections coming from your app instances.

But when elastic beanstalk brings up a new instance it assigns it a random IP address. The script below solves this, by triggering a lambda function with a Cloudwatch rule when a new EC2 instance is launched.

Note: This script assumes you have a pool of elastic IPs – it doesn't provision them.

1. Create a Cloudwatch Rule

Create a cloudwatch rule that runs when an EC2 instance transitions to a "running" state, and have it trigger a lambda function.

2. Create the Lambda Function

Paste the following code into a lambda function:

const AWS = require('aws-sdk');
const ec2 = new AWS.EC2();
const PROD_ENV_NAME = 'my-prod-env-name';

// Example Event
// {
//   "version": "0",
//   "id": "ee376907-2647-4179-9203-343cfb3017a4",
//   "detail-type": "EC2 Instance State-change Notification",
//   "source": "aws.ec2",
//   "account": "123456789012",
//   "time": "2015-11-11T21:30:34Z",
//   "region": "us-east-1",
//   "resources": [
//     "arn:aws:ec2:us-east-1:123456789012:instance/i-abcd1111"
//   ],
//   "detail": {
//     "instance-id": "i-abcd1111",
//     "state": "running"
//   }
// }

exports.handler = async (event) => {
  console.log("EVENT:", event);

  // The newly launched instance ID.
  const instanceId = event.detail['instance-id'];

  // Fetch info about the newly launched instance
  const result = await ec2.describeInstances({
    Filters: [ { Name: "instance-id", Values: [instanceId] } ]
  }).promise()

  // The instance details are buried in this object
  const instance = result.Reservations[0].Instances[0];
  const isAttached = instance.NetworkInterfaces.find(int => int.Association.IpOwnerId !== 'amazon');

  // Bail if the instance is already attached to another EIP
  if (isAttached) {
    console.log("This instance is already assigned to an elastic IP")
    return { statusCode: 200, body: '' }
  }

  // In elastic beanstalk, the instance name gets assigned to the enviroment name.
  // There is also an environment name tag, which could be used here.
  const name = instance.Tags.find(t => t.Key === 'Name').Value;

  // Only assign EIPs to production instances
  if (name !== PROD_ENV_NAME) {
    console.log('Not a production instance. Not assigning. Instance name:', name)
    return { statusCode: 200, body: ''}
  }

  // Get a list of elastic IP addresses
  const addresses = await ec2.describeAddresses().promise();

  // Filter out addresses already assigned to instances
  const availableAddresses = addresses.Addresses.filter(a => !a.NetworkInterfaceId);

  // Raise an error if we have no more available IP addresses
  if (availableAddresses.length === 0) {
    console.log("ERROR: no available ip addresses");
    return { statusCode: 400, body: JSON.stringify("ERROR: no available ip addresses") }
  }

  const firstAvail = availableAddresses[0]
  try {
    // Associate the instance to the address
    const result = await ec2.associateAddress({
      AllocationId: firstAvail.AllocationId,
      InstanceId: instanceId
    }).promise();

    console.log('allocation result', result)

    return { statusCode: 200, body: JSON.stringify('Associated IP address.') };
  } catch (err) {
      console.log("ERROR: ", err);
  }
};

Whenever your elastic beanstalk environment launches a new instance, the above lambda function will get hit, and if there's an available elastic IP address, it will assign it to the new instance.

Graphql Rails Generators

Alex Sharp 🛠sharesecret.co — Tue, 03 Dec 2019 18:53:32 +0000

I recently published some helpful generators for working with graphql in rails. These generators look at your ActiveRecord models and generate model types, input types, and mutations from your ActiveRecord model schema. Enjoy.

https://github.com/ajsharp/graphql-rails-generators

Today’s rails security update in plain english

Alex Sharp 🛠sharesecret.co — Thu, 14 Mar 2019 02:31:33 +0000

Cross-posted from the ShareSecret blog.

Earlier today the Rails team pushed new versions to patch three security vulnerabilities:

CVE-2019-5418: Action view file content disclosure
CVE-2019-5419: Action view Denial of Service (DOS)
CVE-2019-5420: Rails development mode Remote code execution (RCE)

I’ve addressed the vulnerabilities below in order of severity.

Note: The rails upgrade guide is a great resource for upgrading your rails app.

🚨 🚨 🚨 🚨 🚨 🔥 Action View Denial of Service (DOS)

Alarm level: Five alarm fire. Patch immediately.

If you’re rendering tempates, you’re almost definitely subject to a DOS attack. This one is really bad. Patch/upgrade immediately.

Using specially crafted headers you can max out the CPU by exploiting the template location code. Rendering templates wrapped in a respond_to block is safe. Otherwise, you’re vulnerable.

Vulnerable:

class UserController < ApplicationController 
  def index 
    render "index" 
  end 
end

Not vulnerable:

class UserController < ApplicationController 
  def index 
    respond_to |format| 
      format.html { render "index" } 
    end 
  end 
end

🚨 🚨 🚨 🚨 🚨 🔥 Action View File Content Disclosure

Alarm level: Five alarm fire. Patch immediately.

By using specially crafted headers, you can view an arbitrary file’s content with if you use render file: 'filename'. Not good.

The good news: if you’re just rendering normal templates, you’re not affected by this vulnerability, though you’re probably affected by the CVE-2019-5419.

🚨 Rails development mode RCE

Alarm level: Not good, but go back to sleep. Fix it in the morning.

Due to how rails generates the secret_key_base in development mode — an MD5 of the app module name — if you know the name of the application, you can figure out the secret key. As long as you don’t have dev mode apps exposed to the public, this isn’t a huge deal, though is still something worth fixing.

☝️ Be sure to check out Sharesecret, which makes it easy to securely share sensitive data.

A simple react date picker component

Alex Sharp 🛠sharesecret.co — Tue, 12 Mar 2019 09:53:53 +0000

None of the existing react date picker components quite fit my requirements (mostly too bloated with too many dependencies), so I decided to see if I could quickly hack one together. This took a couple of hours, but should meet my needs. Most importantly, it's tiny (~100 LOC) and free of the complexity of many of the existing solutions.

Cross-posted from ajsharp.com/a-simple-react-date-picker-component/.

Disabling specific eslint rules

Alex Sharp 🛠sharesecret.co — Mon, 04 Mar 2019 08:10:58 +0000

The create react app eslint presets come with a few rules that are slightly annoying. One of them is jsx-a11y/href-no-hash, which makes sure you don’t add an tag without a valid http address for the href property.
To ignore this, add a .eslintrc.js file at the project root with the following:

module.exports = {
  "parser": "babel-eslint",
  "env": {
    "browser": true
  },
  "rules": {
    "jsx-a11y/anchor-is-valid": "off"
  }
}

Then make sure to reload the vscode window.

Full eslint config file documentation here.

☝️ Be sure to check out Sharesecret, which makes it easy to securely share sensitive data.

☝️Redux dev tools are a thing and they're incredible

Alex Sharp 🛠sharesecret.co — Fri, 22 Feb 2019 06:42:07 +0000

Today I discovered -- in the spirit of the fantastic React Dev Tools -- there exists an equally if not more useful Redux Dev Tools.

A few of the things you get:

An inspectable timeline of every action dispatched by the store
A state diff for each action. e.g. How each action changed the state object
A full state inspector
A test generator

Wow. Just, wow.

☝️ Be sure to check out Sharesecret, which makes it easy to securely share sensitive data.

The Hilarious Fragility of NLP APIs

Alex Sharp 🛠sharesecret.co — Thu, 21 Feb 2019 19:42:34 +0000

Recently I've been working on functionality in Follow Reset that requires machine learning and natural language processing, so I've been experimenting with two well-known NLP APIs: AWS Comprehend and Google Natural Language. While my primary interest in these API's is for their custom modeling capabilities, I was curious to see what kind of quick results I could get from their basic entity recognition and categorization functionality.

The high-level product goal is simple: use Twitter bios to extract and detect high-level categorical information about people.

My main test case is the profile description of Joe Rogan, a very well-known comedian and podcaster with 4.68M Twitter followers (as of writing).

My (naive) hope was that these APIs would be able to extract from this description that this person is Joe Rogan, who is a comedian.

The results were...uh...surprising, to say the least.

😕 Initial (odd) results

I used Joe's Twitter bio as the input into both APIs:

Stand up comic/mixed martial arts fanatic/psychedelic adventurer Host of The Joe Rogan Experience #FreakParty http://www.facebook.com/JOEROGAN

AWS Comprehend recognizes Joe Rogan as a person. Good start. AWS has a feature called key phrase extraction, that unfortunately in this case, doesn't add much context and is in general pretty useless here.

Google, on the other hand, doesn't actually recognize "Joe Rogan" as a person, though it identifies both "Host" and "mixed martial arts fanatic" as a Person entity. Odd.

While Comprehend's entity results tend to be more factual, (Joe Rogan is a person) Google's results attempt to provide context, identifying direct contextual entities in his bio like "adventurer", as well as a Wikipedia article about the Joe Rogan Experience - his highly popular podcast - context not contained in the bio.

One thing that jumps out here is that while Google identifies categorical context -this is probably the description of a comic - it is unable to properly identify the Person entity to which the comic category refers. Considering Google's ability to detect and suggest context that is external to the input string, it's a bit of a head-scratcher that it fails to correctly identify the Person entity.

🔑 Changing the Inputs, Content is Key?

What happens if we change the name from Joe Rogan to Alex Sharp?

Here's the updated input string:

Stand up comic/mixed martial arts fanatic/psychedelic adventurer Host of The Alex Sharp Experience #FreakParty http://www.facebook.com/AlexSharp

Amazon's still sees Alex Sharp as a person entity. Cool. I am that. 🙋‍♂️

Google's results are…unexpected. Somehow, Google is more confident that Alex Sharp is a Comic than Joe Rogan. Uh, sure 😂

🙃 joe rogan is not Joe Rogan?

What happens if you de-capitalize Joe Rogan to joe rogan? When we do this, Amazon no longer recognizes the person entity. Google still thinks we're probably talking about a comic, but not by much.

What happens if we substitute the name for a fictional character, or if we change the capitalization of some of the words around the name? Here I've changed the name from Alex Sharp to Ronald McDonald.

Both APIs seem to have a much easier time recognizing a Person entity if it's not surrounded by other capitalized words.

Amazon recognizes both forms but has a much higher confidence level when Ronald McDonald is not surrounded by a capitalized word.

Google's results are more stark, failing to recognize a Person entity at all if the term is surrounded by other capitalized nouns: it recognizes the whole phrase Ronald McDonald Experience as "Other". It thinks it's something, but it doesn't know what.

While these results make logical sense to a layman - capitalized words are often proper nouns - it's a bit disappointing that these products rely on such basic and fragile grammar rules.

☝️Massively Unqualified Advice to Amazon & Google

It seems to me (enormous caveat: I'm a complete ML/AI amateur) that the person entity recognition issue could be improved by identifying names based on whether they match (or don't match) dictionary words. Neither the words joe or rogan are dictionary words, so can we reasonably assume, with mild confidence, that paired together, joe rogan might be someone's name? I dunno, I'm way beyond swimming in the deep end of a pool I barely understand 🤷🏻‍♂️.

🧙‍♀️🚫🦄 No Magic Here

As we can see, these APIs can be incredibly fragile, often reacting in odd and unexpected ways to tiny changes in input.

They seem to operate around fairly rudimentary grammatical "rules" - proper nouns must be capitalized, proper nouns must exist on their own - which are in many cases incredibly fragile, especially considering that many of us aren't exactly writing on the internet and social media in academically sanctioned grammar. Change an uppercase letter to lowercase and the whole thing breaks, and poor Joe Rogan is robbed of his personhood. Sad. 😟

Outside of this largely underwhelming but head-scratching exercise in curiosity, there is some truly mind-blowing work happening in deep learning powered machine learning and NLP. Unfortunately, AWS Comprehend and Google AutoML feel more like the training wheels version of a technology that can literally type words before we've thought of them, beat humans at complex strategy games, drive cars, and more. Unfortunately, these APIs are pretty underwhelming for anything other than basic grammar categorization and sentiment analysis (not covered here).

🙏 Thanks for reading. If you're interested in learning more about the product that inspired the work for this post, check out and subscribe to Follow Reset, which will soon make it very easy to clean up your Twitter feed.

React CRA + Netlify = 💯❤

Alex Sharp 🛠sharesecret.co — Mon, 28 Jan 2019 18:13:18 +0000

I've been shipping static frontend to s3/cloudfront for a while, and I finally gave Netlify a chance and wow is it easy. You connect your github repo, give it a command to run and done. They handle all the cloudfront / CDN stuff behind the scenes that I used to have to configure manually.

See here for more.

☝️ Be sure to check out Sharesecret, which makes it easy to securely share sensitive data.

Elasticsearch full-text queries explained, for humans 🤓

Alex Sharp 🛠sharesecret.co — Sat, 26 Jan 2019 01:24:21 +0000

ElasticSearch Full-text query Docs

ES has different query types. A quick summary:

match - standard full text query
match_phrase - phrase matching. like when you put a term in quotes on google.
match_phrase_prefix - poor man’s autocomplete.
multi_match - Multi-field match.
common (not covered here) - Takes stopwords (e.g. “the”) into account.
query_string (not covered here) - Expert mode. Can use AND|OR|NOT and multi-field search in a query string.
simple_query_string (not covered here) - Simpler “more robust” for exposing to end-users. (how is simple more robust? seems counter-intuitive. no one knows.)

`match` - standard full text query

Docs

By default does near-exact word equality matching.

When to use: If you're doing full-text search on a large text field, such as the body of this post.

Good default for matching a word or phrase in a block of text, e.g. searching in a large text field, description field, etc, such as searching for a phrase in someone’s Twitter bio.

Not good for partial matches of a single word or phrase, such as a partial match of someone’s Twitter username (see fuzziness below).

Fuzzy matching
Allows fuzzy matching (e.g. typo tolerance) by specifying a levenstein edit distance, number of one character changes needed to make one string match another string. Basically, it’s the number of typos you can make and still match the string.

Example: Your target match data is a field with the value “ajsharp”. With the fuzziness param set to 1, you can type “jjsharp” and it will match “ajsharp”. But if you type “jssharp” it won’t match, unless you increase fuzziness to 2.

`match_phrase` - for matching exact phrases

Docs

When to use: If you need to be specific about whole phrase searches, or want to enable this functionality for your users.

Let’s say we have two twitter profiles indexed. Document A has the phrase “co-founder” in the bio, and Document B just has the word “founder”.

If you search “founder” using match_phrase, both documents will match.
If you search “co-founder” using match_phrase, only Document A will match. However, had you done this query using match, both Document A and Document B would match the query “co-founder”

`match_phrase_prefix` - poor man’s autocomplete

Docs

When to use: Only you need to match a single field. For many use-cases multi_match feels far more useful.

From the ES docs:

Consider the query string quick brown f. This query works by creating a phrase query out of quick and brown (i.e. the term quick must exist and must be followed by the term brown). Then it looks at the sorted term dictionary to find the first 50 terms that begin with f, and adds these terms to the phrase query.

Can be enhanced/improved upon with Completion Suggester

`multi_match` - multi-field match 🎉🎉🎉

Allows you to search for the same string in multiple fields.

When to use: In many cases this is probably what you need if you're doing anything auto-completey (note: there are probably many uses cases this is great for but my primary one when I wrote this post was autocomplete, so 🤷🏻‍♂️).

Multi match queries can have a type:

best_fields. Default. Match any field, but uses the _score from the best field. Docs
most_fields - Matches any field and combines the score
cross_fields - Treats fields with same analyzer as if one big field.
phrase - Uses match_phrase
🎉🎉🎉 phrase_prefix - Uses match_phrase_prefix on each field and combines the _score. Allows multi-field autocomplete 💥💥💥

Field Boosting
Fields can be boosted by using the ^ followed by number. Assume twitter profiles with screen_name and name fields. If you do a query like this, the screen_name will be three times as important as the name field in the ranking:

{
  "query": {
    "multi_match" : {
      "query" : “ajsharp”,
      "fields" : [ “screen_name^3", “name” ] 
    }
  }
}

☝️ Be sure to check out Sharesecret, which makes it easy to securely share sensitive data.

🚫😩 An array of react refs

Alex Sharp 🛠sharesecret.co — Fri, 25 Jan 2019 04:20:07 +0000

Apparently you can't store React refs in an array. For some reason they get wiped out, so if you need to store a collection of refs, you have to do something like this (forgive me lord, for I hath sinned):

import React from 'react'

const collection = ["label 1", "label 2"]

class SinFactory extends React.Component {
  constructor(props) {
    super(props)
    this.ref0 = React.createRef()
    this.ref1 = React.createRef()
  }

  render() {
    return (
      <div>
        {collection.map((label, i) => {
          return <div key={label} 
            ref={this[`ref${i}`]}>{label}
          </div>
        })}
      </div>
    )
  }
}

It's truly filthy, but it works.

☝️ Be sure to check out Sharesecret, which makes it easy to securely share sensitive data.

DEV Community: Alex Sharp 🛠sharesecret.co

JSON Parsing with Go Embedded Structs

A Disabled React Component

Automatically Assign Elastic IPs to Elastic Beanstalk Instances

1. Create a Cloudwatch Rule

2. Create the Lambda Function

Graphql Rails Generators

Today’s rails security update in plain english

🚨 🚨 🚨 🚨 🚨 🔥 Action View Denial of Service (DOS)

🚨 🚨 🚨 🚨 🚨 🔥 Action View File Content Disclosure

🚨 Rails development mode RCE

A simple react date picker component

Disabling specific eslint rules

☝️Redux dev tools are a thing and they're incredible

The Hilarious Fragility of NLP APIs

😕 Initial (odd) results

🔑 Changing the Inputs, Content is Key?

🙃 joe rogan is not Joe Rogan?

☝️Massively Unqualified Advice to Amazon & Google

🧙‍♀️🚫🦄 No Magic Here

React CRA + Netlify = 💯❤

Elasticsearch full-text queries explained, for humans 🤓

match - standard full text query

match_phrase - for matching exact phrases

match_phrase_prefix - poor man’s autocomplete

multi_match - multi-field match 🎉🎉🎉

🚫😩 An array of react refs

`match` - standard full text query

`match_phrase` - for matching exact phrases

`match_phrase_prefix` - poor man’s autocomplete

`multi_match` - multi-field match 🎉🎉🎉