DEV Community: Glen Bray

Build your own Australian Address Search API

Glen Bray — Sun, 07 Jun 2020 10:06:52 +0000

There are multiple services available that offer API's for Austrlian address autocomplete that also have generous free daily limits. But, there may be times when those limits aren't enough and the costs of using the service may not be viable. An example of a time when this was useful for me can viewed here.

Lets look at whats involved in creating our own Australian address search API using the GNAF dataset.

We'll be doing the following:

Run the gnaf-loader to import address data into a separate postgres DB.
Setup a new rails api project.
Use the new multi-database feature to connect to the GNAF DB in our app.
Sync to Elasticsearch.
Create 2 endpoints for address autocomplete and reverse geolocation lookup.

Import Australian Addresses with the gnaf-loader

Clone the gnaf-loader project.

$ git clone git@github.com:minus34/gnaf-loader.git

The gnaf-loader has a [few options]https://github.com/minus34/gnaf-loader#process) to import the data. Choose whichever option you prefer to load the data into Postgres. This article will cover loading the data with docker.

From your terminal change directory to the repo you've just cloned gnaf-loader
Create a data directory in the root of the repo mkdir data.
Download the PSMA GNAF and ESRI shape file as mentioned from the readme of gnaf-loader.
Extract both and move directories into the data directory.
Before we build the address database let's update the dockerfile and open some ports. The change made can be viewed here.
Run docker-compose up to build and run the containers.

Once the containers start running, this will start the import process. This will take a while to run. While we wait, let's set up the API. Our API will use Elasticsearch so we'll set that up first. With mac and homebrew this can be done with:

$ brew install Elasticsearch
$ brew services start Elasticsearch

Create a new rails project

$ rails new australian_address_api --api --database=postgresql`

Configure your database.yml

development:
  primary:
    <<: *default
    database: australian_address_api_development
  gnaf:
    <<: *default
    database: gnaf

This setup the DB.

$ rails db:setup

We'll use the searchkick gem to search and index records in Elasticsearch. So lets add that to the gemfile and bundle install.

Prepare to sync to Elasticsearch

There are a few things we need to do before we start syncing records to Elasticsearch. Firstly, we'll create an Address model and have it connect to our GNAF DB. We'll use the new rails 6 multiple database feature to have this model connect to our GNAF DB. This will enable us to separate the GNAF database from our API DB. Here's the code to get that in place.

class Address < ApplicationRecord
  connects_to database: {reading: :gnaf, writing: :gnaf}

  self.primary_key = "gid"
  self.table_name = "gnaf_202005.addresses"
end

Now that we can connect to the GNAF DB, lets setup searchkick.

class Address < ApplicationRecord
  # ...

  STREET_SYNONYMS = [
    ['street','st'],
    ['terrace','tce'],
    ['road','rd'],
    ['boulevard','bvd'],
    ['close','cl'],
    ['crest','crst'],
    ['drive','dr'],
    ['avenue',' av'],
    ['highway',' hwy']
  ]

  searchkick default_fields: [:full_address],
      word_start: [:full_address],
      synonyms: STREET_SYNONYMS,
      locations: [:location]

  scope :search_import, -> { where("confidence > 0") }

  def should_index?
    confidence > 0
  end

  def full_address
    [address.titlecase, locality_name.titlecase, state, postcode].join(' ')
  end

  def search_data
    {
      full_address: full_address.downcase,
      suburb: locality_name.downcase,
      state: state.downcase,
      postcode: postcode,
      location: {
        lat: latitude.to_s,
        lon: longitude.to_s
      }
    }
  end
end

The search_data method is used to send the data for a record to Elasticsearch for what we'd like to make searchable. We'll only be using the full_address and location fields in this article.

The search_import and should_index? methods are used to only sync records matching the specified condition. More information on what the confidence score means here.

We'll want to handle abbreviations for certain words within the address e.g street -> st. I've created a synonyms constant for those mappings which are passed to searchkick.

Sync to elasticsearch

Now with that in place, let's start syncing records to Elasticsearch. You can do this with Address.reindex from the rails console. This will sync around 11 - 14 million records and will take a while to do. You can do this async to speed it up, but that is out of the scope of this article. See the searchkick readme for more info.

When syncing finishes, we can test search in the rails console:

pry(main)> Address.search("38a wentworth rd vaucluse").first

Address Search (168.3ms)  addresses_development/_search {"query":{"bool":{"should":[{"dis_max":{"queries":[{"match":{"full_address.analyzed":{"query":"38a Wentworth Road Vaucluse NSW 2030","boost":10,"operator":"and","analyzer":"searchkick_search"}}},{"match":{"full_address.analyzed":{"query":"38a Wentworth Road Vaucluse NSW 2030","boost":10,"operator":"and","analyzer":"searchkick_search2"}}},{"match":{"full_address.analyzed":{"query":"38a Wentworth Road Vaucluse NSW 2030","boost":1,"operator":"and","analyzer":"searchkick_search","fuzziness":1,"prefix_length":0,"max_expansions":3,"fuzzy_transpositions":true}}},{"match":{"full_address.analyzed":{"query":"38a Wentworth Road Vaucluse NSW 2030","boost":1,"operator":"and","analyzer":"searchkick_search2","fuzziness":1,"prefix_length":0,"max_expansions":3,"fuzzy_transpositions":true}}}]}}]}},"timeout":"11s","_source":false,"size":10000}
Address Load (1.8ms)  SELECT "gnaf_202005"."addresses".* FROM "gnaf_202005"."addresses" WHERE "gnaf_202005"."addresses"."gid" = $1  [["gid", 12706313]]

=> #<Address:0x00007f827a068250
 gid: 12706313,
 gnaf_pid: "GANSW710434263",
 street_locality_pid: "NSW2925230",
 locality_pid: "NSW4107",
 alias_principal: "P",
 primary_secondary: nil,
 building_name: nil,
 lot_number: nil,
 flat_number: nil,
 level_number: nil,
 number_first: "38A",
 number_last: nil,
 street_name: "WENTWORTH",
 street_type: "ROAD",
 street_suffix: nil,
 address: "38A WENTWORTH ROAD",
 locality_name: "VAUCLUSE",
 postcode: "2030",
 state: "NSW",
 locality_postcode: "2030",
 confidence: 2,
 legal_parcel_id: "381//DP1061794",
 mb_2011_code: 10892260000,
 mb_2016_code: 10892260000,
 latitude: -0.3385606636e2,
 longitude: 0.15126996495e3,
 geocode_type: "PROPERTY CENTROID",
 reliability: 2,
 geom: "0101000020BB1000001FEA888DA3E86240F0B31D9593ED40C0">

Address autocomplete

While we wait for the sync to finish let's implement autocomplete address endpoint.

# app/controllers/address_autocomplete_controller.rb
Rails.application.routes.draw do
  get "/address/autocomplete", controller: :address_autocomplete, action: :index
end

# app/controllers/address_autocomplete_controller.rb
class AddressAutocompleteController < ActionController::API
  def index
    @addresses = Address.search(params[:q], match: :word_start)
  end
end

With the jbuilder gem, we can create a few templates that will generate our JSON. When setting up a rails project this gem is included in the gemfile but commented out. Uncomment it and run bundle install.

# app/views/addresses/_address.jbuilder
json.address address["address"]
json.lot_number address["lot_number"]
json.flat_number address["flat_number"]
json.level_number address["level_number"]
json.number_first address["number_first"]
json.number_last address["number_last"]
json.street_name address["street_name"]
json.street_type address["street_type"]
json.street_suffix address["street_suffix"]
json.suburb address["locality_name"]
json.postcode address["postcode"]
json.state address["state"]
json.longitude address["longitude"]
json.latitude address["latitude"]

The above is a shared partial that will be used by both autocomplete and reverse geolocation.

# app/views/address_autocomplete/index.jbuilder
json.partial! 'addresses/address', collection: @addresses, as: :address

Reverse geolocation

Another feature that we'll implement is the ability to search for addresses near a given
longitude and latitude.

Let's update the address model to support reverse geocode searches.

Address
  # ...

  def self.reverse_geocode(longitude, latitude, within)
    within = 1 if within.blank? || within <= 0

    Address.search(
      where: {
        location: {
          near: {
            lon: longitude.to_f,
            lat: latitude.to_f
          },
          within: "#{within}m",
        }
      }
    )
  end
end

Finally, to finish this off - the route, controller and view.

# app/controllers/address_autocomplete_controller.rb
Rails.application.routes.draw do
  get "/coordinates/reversegeocode", controller: :reverse_geocode, action: :index
end

class ReverseGeocodeController < ActionController::API
  def index
    @addresses = Address.reverse_geocode(params[:lng], params[:lat], params[:within])
  end
end

Our API will accept the params longitude, latitude and within. The within params accepts an integer. It allows us to filter addresses within a distance in meters from the longitude and latitude.

# app/views/reverse_geocode/index.jbuilder
json.partial! 'addresses/address', collection: @addresses, as: :address

That completes our minimal Australian address API. As mentioned above, there are API's available that have a daily free limit and are fairly cheap. Depending on your situation it might be better to just use those to save time.

Elasticsearch Search Parser Part 2

Glen Bray — Mon, 16 Mar 2020 21:14:56 +0000

The code for this article can be found here.

Continuing on from PART 1 we'll look to finish off the remaining features of our gem.

Part 1

Term search - term
Phrase search "this is a phrase"
AND condition cat dog or cat AND dog

Part 2

OR Condition cat OR dog
Not filter -term
Grouped search (cat dog) OR (crocodile fish)

OR condition

The image below should give you an idea of how a tree may look like after Parslet rules have been applied with our new OR operator.

As we did in part 1 we'll get started with an integration test.

RSpec.describe ElasticParser do
  describe ".parse" do
    #...

    describe "OR query" do
      let(:search_terms) { ['supplier', 'dog'] }

      let(:expected) do
        {
          :query => {
            :bool => {
              :should => search_terms.map do |word|
                {
                  :bool => {
                    :minimum_should_match => 1,
                    :should => {
                      :multi_match => {
                        :fields => ElasticParser::FIELDS,
                        :query => word
                      }
                    }
                  }
                }
              end
            }
          }
        }
      end

      context "with OR operator" do
        let(:query) { "supplier OR dog"}

        it "returns match phrase query" do
          expect(subject).to eq(expected)
        end
      end
    end
  end
end

The parser spec can be updated to handle the OR operator.

RSpec.describe ElasticParser::Parser do
  #...

  describe '#or_op' do
    it 'parses OR' do
      expect(subject.or_op).to parse(' OR ')
    end
  end

  describe '#or_condition' do
    it 'parses with OR' do
      expect(subject.or_condition).to parse('a OR b')
    end
  end
end

To get our parser spec working we'll make some changes to the parser class:

Update term rule to ignore the OR operator.
Create a rule to handle the OR operator.
Create a rule to match an OR condition.
Update query rule to call or_condition

module ElasticParser
  class Parser < Parslet::Parser
    #...

    rule(:term) do
      str("OR").absent? >> match('[^\s"]').repeat(1)
    end

    rule(:or_op)   { (space >> str("OR") >> space) }

    rule(:or_condition) do
      (
        and_condition.as(:left) >> or_op >> or_condition.as(:right)
      ).as(:or) | and_condition
    end

    rule(:query) { or_condition.as(:query) }
  end
end

This may be a bit confusing if you've not done much recursion in a while (or at all).

When the or_condition is evaluated it will attempt to build a left subtree. It will first attempt to match an AND operator before attempting to match the OR operator. If it's unable to match the OR or AND operators, then it will fall back and match the value rule (term or phrase).

In the middle of the or_condition rule, it will then attempt to match an OR operator. Then for the right subtree, it will call itself and do what we did to the left subtree.

Let's make another update the transformer class to handle the OR operator.

module ElasticParser
  class Transformer < Parslet::Transform
    rule(or: { left: subtree(:left), right: subtree(:right) }) do
      node = Nodes::OperatorNode.new(:or, left, right)
      left.parent = node if left
      right.parent = node if right
    end
  end
end

Finally we'll update the OperatorNode class to return the correct Elasticsearch operator for the OR operator.

module ElasticParser::Nodes
  class OperatorNode < ElasticNode
    def to_elastic_op(data)
      case data
      when :and
        :must
      when :or
        :should
      else
        raise "Unknown operator: #{operator}"
      end
    end

    #...
  end
end

Now when we run our specs again:

NOT operator

The NOT operator is a special case. It differs from the other operator nodes as it is only applied to the term or phrase immediately after the operator. When we set the attributes on the node class, we're only going to set the left attribute and ignore the right.

The image below demonstrates the tree the parser will build for a given expression.

We'll add an integration test to handle the not operator.

RSpec.describe ElasticParser do
  describe ".parse" do
    #...

    describe "NOT query" do
      let(:query) { "-dog"}

      let(:expected) do
        {
          :query => {
            :bool => {
              :must_not => {
                :bool => {
                  :minimum_should_match => 1,
                  :should => {
                    :multi_match => {
                      :fields => ElasticParser::FIELDS,
                      :query => 'dog'
                    }
                  }
                }
              }
            }
          }
        }
      end

      context "with NOT operator" do
        it "returns match phrase query" do
          expect(subject).to eq(expected)
        end
      end
    end
  end
end

We'll also update to the parser specs for the not operator.

RSpec.describe ElasticParser::Parser do
  #...

  describe '#not_op' do
    it 'parses -' do
      expect(subject.not_op).to parse('-')
    end
  end

  describe '#not_condition' do
    it 'parses with -' do
      expect(subject.or_condition).to parse('a OR b')
    end
  end
end

Now let's make the changes to the parser class to get the parser specs working.

require "parslet"

module ElasticParser
  class Parser < Parslet::Parser
    #...

    rule(:not_op)   { str('-') }

    rule(:not_condition) do
      (
        not_op >> value.as(:left) >> space.maybe
      ).as(:not) | value
    end

    rule(:and_condition) do
      (
        not_condition.as(:left) >> and_op >> and_condition.as(:right)
      ).as(:and) | not_condition
    end
  end
end

We'll create a new node to handle NOT operations. The NotOperatorNode will extend the OperatorNode. It overrides the to_query method and only generates a left subtree when building the Elasticsearch query.

module ElasticParser::Nodes
  class NotOperatorNode < OperatorNode
    def to_query
      { bool: { must_not: left.to_query } }
    end
  end
end

Now let's update the transformer and add a new rule to handle the NOT operator.

module ElasticParser
  class Transformer < Parslet::Transform
    rule(not: { left: subtree(:left) }) do
      node = Nodes::NotOperator.new(:not, left)
      left.parent = node if left
    end
  end
end

Let's run our specs again

Grouped Search

The final feature we'll add is the abillity change the order or operations with parenthesis which works the same as how you would expect it to in maths. Anything within the parenthesis will be evaluated first.

Let's implement our final integration test.

RSpec.describe ElasticParser do
  describe ".parse" do
    #...

    describe "Grouped query" do
      let(:query) { "(a b) OR (c (d e))" }

      let(:expected) do
        {
          :query => {
            :bool => {
              :should => [{
                :bool => {
                  :must => [{
                    :bool => {
                      :minimum_should_match => 1,
                      :should => {
                        :multi_match => {
                          :fields => ElasticParser::FIELDS,
                          :query => "a"
                        }
                      }
                    }
                  }, {
                    :bool => {
                      :minimum_should_match => 1,
                      :should => {
                        :multi_match => {
                          :fields => ElasticParser::FIELDS,
                          :query => "b"
                        }
                      }
                    }
                  }]
                }
              }, {
                :bool => {
                  :must => [{
                    :bool => {
                      :minimum_should_match => 1,
                      :should => {
                        :multi_match => {
                          :fields => ElasticParser::FIELDS,
                          :query => "c"
                        }
                      }
                    }
                  }, {
                    :bool => {
                      :must => [{
                        :bool => {
                          :minimum_should_match => 1,
                          :should => {
                            :multi_match => {
                              :fields => ElasticParser::FIELDS,
                              :query => "d"
                            }
                          }
                        }
                      }, {
                        :bool => {
                          :minimum_should_match => 1,
                          :should => {
                            :multi_match => {
                              :fields => ElasticParser::FIELDS,
                              :query => "e"
                            }
                          }
                        }
                      }]
                    }
                  }]
                }
              }]
            }
          }
        }
      end

      it 'generates a nested query' do
        expect(subject).to eq(expected)
      end
    end
  end
end

Now for the final parser specs.

RSpec.describe ElasticParser::Parser do
  #...

  describe '#group' do
    it 'parses terms within parentheses' do
      expect(subject.group).to parse('(a b)')
    end

    it 'parses phrase within parentheses' do
      expect(subject.group).to parse('(a "b c")')
    end

    it 'parses more complicated nesting' do
      expect(subject.group).to parse('(b (c (d e)))')
    end
  end
end

The remaining changes we need to make to the parser are:

Create rules for parenthesis
Update the term rule to ignore the parenthesis
Create a group rule that will match a group within an expression, otherwise match on value rule
Update the not_condition rule to fall back to the group rule if no match made

require "parslet"

module ElasticParser
  class Parser < Parslet::Parser
    #...

    rule(:lparen)   { str('(') }
    rule(:rparen)   { str(')') }

    rule(:term) do
      str("OR").absent? >> match('[^\s"()-]').repeat(1)
    end

    rule(:group) { (lparen >> or_condition >> rparen) | value }

    rule(:not_condition) do
      (
        not_op >> value.as(:left) >> space.maybe
      ).as(:not) | group
    end
  end
end

That's all we need to do for that one, so let's run the specs again.

All specs are passing and we've implemented all the features that we want.

How does the Elasticsearch query actually get generated?

Up until now, I have not gone into the details how we actually generate the Elasticsearch query from the tree that we generate with our node classes. This works by applying left to right recursion. Each node implements a to_query method.

We store the root node as a variable within the ElasticTree class which also implements a method to_query. If a node within the tree is an OperatorNode the queries that it generates will call the left and right attributes respectively. We evaluate the left branch first, then it will recursively go through each node until it gets to a leaf node.

The leaf node is the end of a branch within the tree that will return a term or phrase. As it traverses through each node it will build a hash the same shape as the tree and this hash that is returned is our Elasticsearch query that a library such as Searchkick will accept. If you're using Searchkick you'll need to use the advanced search feature.

References

Elasticsearch Search Parser Part 1

Glen Bray — Mon, 16 Mar 2020 21:14:49 +0000

Out of the box, Elasticsearch provides its own query parser. But what if you wanted something similar to googles search functionality. In our case, we don't need all the features that come with Elasticsearch and we'd prefer to restrict functionality. So our plan is to build our own gem that can take a search expression then transform that and return an Elasticsearch query.

The code for this article can be found here.

I've split building this gem into two articles. The features we will work on are listed in part one below.

Part 1

Term search - term
Phrase search "this is a phrase"
AND condition cat dog or cat AND dog

Part 2

OR Condition cat OR dog
Not filter -term
Grouped search (cat dog) OR (crocodile fish)

Before we start writing any code for a feature, we will start with tests. So we'll build this test first. Let's get started by creating the gem.

bundle gem elastic_parser

We're using the Parslet gem to handle parsing so let's also update the elastic_parser.gemspec to add that dependency.

spec.add_dependency "parslet"

Term search

As mentioned above, we'll do this test first. So the first thing we'll do is create an integration test to build a query from a simple term expression.

The first thing we'll as we're doing this test first is to create an integration test.

RSpec.describe ElasticParser do
  describe ".parse" do
    subject { ElasticParser.parse(query) }

    describe "simple query" do
      let(:query) { "word" }

      it "returns multi match query" do
        expected = {
          :query => {
            :bool => {
              :minimum_should_match => 1,
              :should => {
                :multi_match => {
                  :fields => ElasticParser::FIELDS,
                  :query => query
                }
              }
            }
          }
        }

        expect(subject).to eq(expected)
      end
    end
  end
end

For this spec, we'll pass in a simple term that is word and we're expecting the result to be an Elasticsearch query. I'll be performing a multi-match query in Elasticsearch against each multiple fields in a doc. That is more specific for my use case.

Let's go ahead and create a spec for our Parser class

require 'parslet/rig/rspec'

RSpec.describe ElasticParser::Parser do
  describe "term" do
    it "parses term" do
      expect(subject.term).to parse("a")
    end
  end
end

To get this spec to pass we'll create the Parser class and implement the term rule which will match any character

require "parslet"

module ElasticParser
  class Parser < Parslet::Parser
    rule(:term) { match('.').repeat(1).as(:term) }
    rule(:query) { term.as(:query) }
    root(:query)

    def self.parse(raw_query)
      new.parse(raw_query)
    rescue Parslet::ParseFailed => e
      puts e.parse_failure_cause.ascii_tree
    end
  end
end

Now that the parser spec works, let's get the integration spec working. What we'll do now is take our parsed term and transform it into an Elasticsearch query.

We'll first start putting together our Transformer class. When the Parser runs it will generate a hash with key-value pairs from the rules that we have defined. We'll use the transformer class to take that hash and transform it into a Binary Tree.

module ElasticParser
  class Transformer < Parslet::Transform
    rule(term: simple(:term)) do
      Nodes::LeafNode.new(term: term.to_s.downcase)
    end

    rule(query: subtree(:query)) { ElasticTree.new(query) }
  end
end

To construct our tree we'll need to set up our node classes. We'll first implement a base node class called ElasticNode here's what it looks like.

module ElasticParser::Nodes
  class ElasticNode
    attr_accessor :left, :right, :parent

    def initialize(data, left = nil, right = nil, parent = nil)
      @data = data
      @left = left
      @right = right
      @parent = parent
    end

    def to_query
      raise "implement me!"
    end
  end
end

You'll notice the left, right, and parent accessors. These will store references to other nodes. As we are implementing a binary tree, we'll restrict the children to a maximum of two branches (left, right).

We'll now implement our LeafNode that extends the ElasticNode which will store terms or phrases that are parsed.

module ElasticParser::Nodes
  class LeafNode < ElasticNode
    def to_query
      _key, value = @data.to_a.flatten

      {
        bool: {
          should: {
            multi_match: {
              query: value,
              fields: ElasticParser::FIELDS
            }
          },
          minimum_should_match: 1
        }
      }
    end
  end
end

Now for the ElasticTree class. This class exists to store the root node and to call the to_query method on the root node.

module ElasticParser
  class ElasticTree
    def initialize(tree)
      @root = tree
    end

    def to_query
      { query: @root.to_query }
    end
  end
end

All we need now to get out first integration test passing is to implement the parse method which is the entry point to the lib.

module ElasticParser
  FIELDS = ["title", "content"]

  def self.parse(query)
    parse_tree = Parser.parse(query)
    elastic_tree = Transformer.new.apply(parse_tree)
    elastic_tree.to_query
  end
end

When we run the specs again the integration test should pass.

Phrase search

Now that we have our first integration test working, let's start extending the features of the library. We'll now be looking into generating phrase searches when a user adds quotations around some terms e.g "this is a phrase".

Let's add our next integration spec

RSpec.describe ElasticParser do
  describe ".parse" do
    #...

    describe "phrase search" do
      let(:query) { '"this is a phrase"' }

      it "returns match phrase query" do
        expected = {
          :query => {
            :bool => {
              :minimum_should_match => 1,
              :should => ElasticParser::FIELDS.map do |field|
                { match_phrase: { field => "this is a phrase" } }
              end
            }
          }
        }

        expect(subject).to eq(expected)
      end
    end
  end
end

We'll also update the parser spec.

RSpec.describe ElasticParser::Parser do
  #...

  describe "#space" do
    it "parses space" do
      expect(subject.space).to parse(" ")
    end
  end

  describe "#space?" do
    it "parses empty string" do
      expect(subject.space?).to parse("")
    end

    it "parses space" do
      expect(subject.space?).to parse(" ")
    end
  end

  describe '#quote' do
    it 'parses quotes' do
      expect(subject.quote).to parse('"')
    end
  end

  describe '#phrase' do
    it 'parses words wrapped in quotes' do
      expect(subject.phrase).to parse('"a b c"')
    end
  end
 end

Now with these specs in place some changes need to be made to the parser to support phrases and get our tests working. The changes I've made to the parser code are listed below.

Updated the term rule to support checking for spaces using regex.
Added a quote rule to check for quotation marks ".
Added a phrase rule that reuses our other rules to check for phrases within a supplied string.
Updated the query rule to return term or phrase.
Added a value rule which will return a term or phrase. The phrase rule uses the term rule so we've moved .as(:term) from the term rule to the value rule.

module ElasticParser
  class Parser < Parslet::Parser
    rule(:space) { str(" ").repeat(1) }
    rule(:space?) { space.maybe }
    rule(:term) { match('[^\s"]').repeat(1) }
    rule(:quote) { str('"') }

    rule(:phrase) do
      (quote >> (term >> space?).repeat.as(:phrase) >> quote) >> space?
    end

    rule(:value) { (term.as(:term) | phrase) }
    rule(:query) { value.as(:query) }

    root(:query)
  end
end

Now our parser specs will now be passing.

Let's update the LeafNode class to handle phrases.

module ElasticParser::Nodes
  class LeafNode < ElasticNode
    def inspect
      { data: @data.inspect }
    end

    def to_query
      key, value = @data.to_a.flatten

      case key
      when :term
        {
          bool: {
            should: {
              multi_match: {
                query: value,
                fields: ElasticParser::FIELDS
              }
            },
            minimum_should_match: 1
          }
        }
      when :phrase
        {
          bool: {
            minimum_should_match: 1,
            :should => ElasticParser::FIELDS.map do |field|
              { match_phrase: { field => value } }
            end
          }
        }
      end
    end
  end
end

You could also refactor this to two separate nodes (TermNode, PhraseNode) to remove the case statement. But we'll just go with the case statement.

We'll add the phrase rule to the transformer class that will transform the phrase hash into a leaf node. We've pass phrase as the key, which the case statement will match in the LeafNode's to_query method and will return the correct Elasticsearch query for a phrase.

module ElasticParser
  class Transformer < Parslet::Transform
    #...

    rule(phrase: simple(:phrase)) do
      Nodes::LeafNode.new(phrase: phrase.to_s.downcase)
    end
  end
end

Now when we run all our specs again everything will pass.

AND condition

Next up we'll work on implementing support for the AND operator. Spaces within a query will also act as an AND operator, with the exception of phrases.

A search expression may look like this supplier "PCB boards". The expectation here is that we would like to perform a search where the content contains the terms supplier AND the phrase PCB boards.

To better visualize how a query is constructed let's have a look at what the parser does with a couple of expressions.

Let's start working on this feature. As we've done before we'll start with an integration test.

RSpec.describe ElasticParser do
  describe ".parse" do
    #...

    describe "AND query" do
      let(:search_terms) { ['supplier', 'dog'] }

      let(:expected) do
        {
          :query => {
            :bool => {
              :must => search_terms.map do |word|
                {
                  :bool => {
                    :minimum_should_match => 1,
                    :should => {
                      :multi_match => {
                        :fields => ElasticParser::FIELDS,
                        :query => word
                      }
                    }
                  }
                }
              end
            }
          }
        }
      end

      context "with space" do
        let(:query) { "supplier dog" }

        it "returns match phrase query" do
          expect(subject).to eq(expected)
        end
      end

      context "with AND operator" do
        let(:query) { "supplier AND dog"}

        it "returns match phrase query" do
          expect(subject).to eq(expected)
        end
      end
    end
  end
end

We'll add a few more parser specs as well.

RSpec.describe ElasticParser::Parser do
  subject { ElasticParser::Parser.new }
  #...

  describe '#and_op' do
    it 'parses AND' do
      expect(subject.and_op).to parse(' AND ')
    end

    it 'parses space when AND is not provided' do
      expect(subject.and_op).to parse(' ')
    end
  end

  describe '#and_condition' do
    it 'parses with space' do
      expect(subject.and_condition).to parse('a b')
    end

    it 'parses with AND' do
      expect(subject.and_condition).to parse('a AND b')
    end
  end
end

Let's update the parser and add a couple of rules to handle the AND operator and generating a rule to parse AND conditions. It gets slightly more complicated as we've also introduced recursion. I'll explain how this works.

Check for a value (term or phrase)
Check for an AND operator
Now we'll recursively recheck

If we look at the and_condition rule you can see that we first check for value (term or phrase). Then a check for an AND operation (spaces that are not apart of phrases are also considered as an AND operator).

require "parslet"

module ElasticParser
  class Parser < Parslet::Parser
    #...

    rule(:and_op)   { ((space >> str('AND') >> space) | space?) }

    rule(:and_condition) do
      (
        value.as(:left) >> and_op >> and_condition.as(:right)
      ).as(:and) | value
    end

    rule(:query) { and_condition.as(:query) }
  end
end

To handle operators, we'll create a new node for our tree called OperatorNode. Its responsibility will handle generating the correct Elasticsearch query for each operator type. For now, we'll have it generate an Elasticsearch must query.


module ElasticParser::Nodes
  class OperatorNode < ElasticNode
    def to_elastic_op(data)
      case data
      when :and
        :must
      else
        raise "Unknown operator: #{operator}"
      end
    end

    def operator
      to_elastic_op(@data)
    end

    def to_query
      { bool: { operator => [left.to_query, right.to_query].flatten } }
    end
  end
end

We'll create a new rule in our transformer that will create our OperatorNode when it matches an and operator.

module ElasticParser
  class Transformer < Parslet::Transform
    #...

    rule(and: { left: subtree(:left), right: subtree(:right) }) do
      node = Nodes::OperatorNode.new(:and, left, right)
      left.parent = node if left
      right.parent = node if right
    end
  end
end

We'll run our specs again and everything will pass.

CONTINUE TO PART 2

References

Extracting addresses from millions of pages with AutoML and Ruby

Glen Bray — Wed, 12 Feb 2020 12:14:58 +0000

Code can be found here

I ran into a problem in the past where I needed to extract Australian addresses from a few million pages. Utilising Google’s AutoML NLP (Natural Language Processing) service we can solve the problem of extracting addresses. I’d then need to somehow take those addresses and make them more useful by splitting them into address parts (street no, street name, postcode, state, etc) and geocoding each address(longitude and latitude).

We could then store the data into Elasticsearch and perform geolocation based searches, allowing users to get the distance from their current location to an address listed on a site or filtering sites based on suburbs, states or postcodes.

The diagram below shows how we go from scraping HTML that contains addresses. To address records that are stored in Postgres and Elasticsearch. I’ll mainly focus on the green boxes.

To give you a better idea of what we are expecting, we’ll be going from raw HTML that may contain an address like this:

<html>
  <body>
    <p>
      Contrary to popular belief, Lorem Ipsum is not simply random text. It has roots in a  piece of classical Latin literature from 45 BC, making it over 2000 years old.
    </p>
    <span>address:</span>
    <span>Unit 2, Marina Mirage, 74 Seaworld Drive, Main Beach QLD 4217</span>
    <span>phone:</span>
    <span>11112222</span>
    ....
  </body>
</html>

Then transforming each address into something like this:

[1] pry(main)> Address.first
=> #<Address:0x00007fc3ce9a4fb8
 id: 1,
 site_id: 104,
 street_no: nil,
 street_name: "Seaworld Drive",
 suburb: "Main Beach",
 state: "QLD",
 postcode: "4217",
 lat: -0.27968262e2,
 lng: 0.153426398e3,
 status: "verified",
 nlp_address: "unit 2, marina mirage, 74 seaworld drive, main beach qld 4217",
 nlp_confidence: 0.99984866e0,
 mappify_confidence: 1,
 created_at: Mon, 10 Feb 2020 12:48:35 UTC +00:00,
 updated_at: Mon, 10 Feb 2020 14:22:23 UTC +00:00>

Scraping sites

You will need to have links to scrape. I won’t provide these but you could buy these online from various places to at least get you started. When scraping you could collect links to other domains that may exist on a page and scrape those as well.

I’ve used Scrapy and deployed it to Scrapy cloud. Scrapy cloud ‘s free plan provides a single process with a one hour run time limit per run. You can run a process as many times as you’d like.

If you need scheduling, Scrapy cloud does not provide it for free. If you want free scheduling you can implement it yourself using their API along with Sidekiq & Sidekiq cron.

Creating training data

This part will be quite tedious but of course, essential. If you were to manually label thousands of addresses, it can take many hours (I know from experience). The prediction model used in the examples of this article has been trained with 2 labels — po_box and address.

Labelling PO box addresses allow us to filter them out when predicting, as they are irrelevant to my requirements. If we don’t teach it the difference, it will return a partial PO box address as an address. This occurs because of the similarity of an address and PO box.

Here are a couple of tools to help annotate your data.

Label Studio — free
Doccano — free
prodigy — paid

At the time I had used Doccano. It didn’t have support to export to Google supported format so I‘ve written my own script to transform the Doccano export file into a format Google AutoML would accept.

I only just found Label Studio, that one looks promising.

There are also services that allow you to outsource this work if it’s within your budget. I can’t advise on this as I just did it myself.

Training a prediction model

We’ll go through the process of taking some training data and creating the prediction model on google cloud.

Setup Google account

Sign up for an account over at https://console.cloud.google.com (you’ll need to add debit/credit card details to activate the free trial)
Create a new project e.g automl-address-extractor

Copy training files to storage

From the storage section of google, console create a new bucket
Upload your annotations file to the bucket
Create a CSV file google_annotations.csv. The contents should contain the location of the annotations file e.g ,gs://bucket-name/google_annotations.jsonl
Upload the CSV file to the bucket the annotations file was uploaded to

Create a google cloud service account

In cloud console, select “IAM & admin” and choose “Service accounts”.
Create a new service account
Select the Role “AutoML Predictor”
Create key and download the JSON key type

The JSON file contains credentials which you can then use to set environment variables in your project required by the googleauth gem. See .env.example for the env var keys.

Setup AutoML

Head over to the natural language area of the console under artificial intelligence
Enable the natural language models API when prompted
Click on AutoML Entity Extraction

Click “New Dataset”
Select the CSV file that was uploaded earlier & import.
Your items screen should look something like this

Go to Train and click the button Start Training. This can take several hours.
Once training is finished, you can test out your model in Test & Use.

Heres an example:

Test prediction API from rails console

Let’s confirm that we can now start predicting addresses with our code within the rails console.

We’ve got an AutoMLClient class that takes a string and returns a prediction from AutoML’s API.

[1] pry(main)> client = AutoMLClient.new
[2] pry(main)> client.predict("1300 111 222 unit 2, marina mirage, 74 seaworld drive, main beach qld 4217 2019 copyright")
=> {"payload"=>
  [{"annotationSpecId"=>"7316645151454003200",
    "displayName"=>"address",
    "textExtraction"=>{"score"=>0.99951684, "textSegment"=>{"startOffset"=>"13", "endOffset"=>"74", "content"=>"unit 2, marina mirage, 74 seaworld drive, main beach qld 4217"}}}]}

As you can see AutoML has managed to extract out the correct address from the string.

Cost of predicting

So far so good right? If you’re only running predictions in the thousands that would be okay. But what if we need to process a million or more addresses? The first thing that comes to mind is cost.

Immediately we can reduce costs substantially simply by avoiding sending entire pages to AutoML. Most of the content on a page will be irrelevant and a page can contain hundreds to thousands of characters. Instead, we can extract snippets from a page where we think an address may exist. I’ll explain how we extract snippets later when we go through the code. But just so you know, a snippet may or may not contain an actual address. That’s why we need to use NLP.

Let’s take a look at some approximate numbers and determine if this can be a viable solution. Because of the way snippet extractions work, the number of characters per snippet is variable. But we can take an average of the snippet length and use that when doing the maths. Heres google pricing page if you’re interested.

At the time of writing the costs are:

We’re more interested in the prediction pricing of text records — $5 per 1000 text records. If a request is 1000 characters or less, it will work out to be 1 text record. 4500 characters would be 5 text records.

Now let’s take a look out our costs currently for predicting addresses from snippets with AutoML.

One snippet per request

Number of snippets: 1 000 000
The average length of a snippet: 110 characters

$5 per 1000 text records = 5 / 1000 = 0.005

Cost: 1 000 000 * 0.005 = $5000

Let’s explore if it’s possible to optimise on cost. By looking at data that has already been collected we know a snippet will have an average length of 110 characters. We also know that google charges per text record of 1000 characters. So what would the costs look like if we could code up a solution to group snippets into blocks of text up to 1000 characters?

Grouping snippets by 1000 characters

1 000 000 * 110 = 110000000 (total number of characters)

Text records: (110000000 / 1000) = 110 000

110 000 * 0.005 = $550

We’ll be saving approximately $4450 USD.

Looks much better (if you have free trial credits), but before we move forward, let’s check the limitations of googles API.

Google allows up to 10000 characters per request. When we group snippets in blocks of 1000 characters, in most cases we won’t get exactly 1000 characters. We could end up with something like 920 or 970. Those missing characters can add up and increase costs over time. So what we’ll do is group snippets in blocks of a maximum of 10000 characters. This will allow us to save a bit more money.

If you’re wondering if AutoML is a good fit for you, it depends on the amount of data you need to process and how much value you can get from that data. If you’re looking to reduce costs further you could implement the entity extraction yourself with spacy. Then build your own prediction API. But that depends on your own personal situation and if the costs of rolling your own solution are worth it.

Google provides a free trial period with $500 credit when setting up a new account. The first 5,000 text records are free during the trial period (5 million characters). That may be enough for you to get a quick MVP out the door, then move on to something like spacy if you need to in the future. There isn’t much stopping you from signing up with another account for more free credits as well. If you’re starting a business, you can also apply for startup credits and get another $3000 credits or more on top.

Extracting addresses with ruby

When we save a page for a site, we store the HTML in our DB. We then strip the HTML and take that result and save it to the content field of a page record. Now we won’t need to worry about HTML when processing.

Extracting snippets

If you look at the snippet extractor code, you’ll see that it will go through each word looking for an Australian state e.g NSW. Once a state has been found, it will extract the surrounding words including the state. This is what we will call the address snippet.

Grouping snippets

Grouping is a bit more complicated. Googles API is expecting a block of text with 10000 characters max. When calling the API, if a prediction is successful it will return the index positions of labels found. We need to take that index position and find the snippet associated with that prediction. The snippet will contain additional information that we require to associate to a site within our DB. So that is the responsibility of the MLContentBuilder class. It will:

join a collection of snippets, and separate with a full stop.
keep track of start and end positions of each snippet
keep track of site id for a snippet
allow us to look up a snippet at an index position

Snippets are separated with a full stop to improve the accuracy of a prediction.

Entity extraction with AutoML NLP (Natural Language Processing)

There is an ExtractAddress service that we’ll use to go from HTML to extracted addresses. When executed it performs the following:

Retrieve from the DB pages for multiple sites
Extract address snippets from pages
Group snippets to a maximum of 10 000 characters for each group
Send grouped snippets to AutoML NLP service to extract addresses
Bulk import into addresses table

Retrieving useful information from an address string

So now we got an address as a string, what now? You may already be familiar with address autocompletion provided by Google or other APIs.

There are several API’s that are Australian address only. I’ve found that those contain more complete data than google. Heres a few Australian addresses only services from a quick search:

Looking at the pricing page, Mappify is the cheapest with 2500 free requests per day. If none of the services is within your budget another option is to build your own address search API using the GNAF dataset. Setting it up is out of the scope of this article, but here are some tips if you’re interested:

Follow the instructions in this repo to load up the GNAF dataset into Postgres
If you decide to use rails as your API, you could use Searchkick to sync to Elasticsearch
Update your model so Searchkick’s callbacks will sync address parts and geolocation data to Elasticsearch
Create a search endpoint and use Searchkick’s search method to perform searches.

All of the Australian address API’s that I’ve found mention that they are using the GNAF dataset.

We’ll keep things simple and use Mappify as I don’t need to be processing millions of snippets for this article. I’ve modified the extracted code to use Mappify for address lookups.

Source for Mappify Client

[1] pry(main)> mc = MappifyClient.new("your-api-key-goes-here")
[2] pry(main)> mc.address_search("unit 2, marina mirage, 74 seaworld drive, main beach qld")
=> {"type"=>"completeAddressRecordArray",
 "result"=>
  [{"buildingName"=>"MARINA MIRAGE",
    "flatNumberPrefix"=>nil,
    "flatNumber"=>2,
    "flatNumberSuffix"=>nil,
    "levelNumber"=>nil,
    "numberFirst"=>74,
    "numberFirstPrefix"=>nil,
    "numberFirstSuffix"=>nil,
    "numberLast"=>nil,
    "numberLastPrefix"=>nil,
    "numberLastSuffix"=>nil,
    "streetName"=>"SEAWORLD",
    "streetType"=>"DRIVE",
    "streetSuffixCode"=>nil,
    "suburb"=>"MAIN BEACH",
    "state"=>"QLD",
    "postCode"=>"4217",
    "location"=>{"lat"=>-27.96826181, "lon"=>153.42639831},
    "streetAddress"=>"Unit 2, Marina Mirage, 74 Seaworld Drive, Main Beach QLD 4217"}],
 "confidence"=>0.9180327868852459}

The SearchAddress service will accept addresses that have been extracted with AutoML then look up the address using Mappify’s autocomplete API. It will then bulk update the address records.

Sidekiq Jobs

I won’t cover the code for both services, just the jobs that handle address extraction.

There are two workers involved in the address extraction process.

First is the scheduled job. It will find sites where addresses haven’t been extracted. Then, in batches, it will enqueue another worker, passing it a list of site ids.

The Job that was enqueued will retrieve sites from the ids and execute the ExtractAddress service.

That’s it, pretty straight forward stuff. I’ve found the most difficult parts are getting the data to train and annotating the data to be used for training.

AutoML can be expensive if you’ve got a lot of data to process. But the free trial can go a long way as well. From the calculations above I can cover the costs of processing a million snippets with the free trial.