DEV Community: Gaël

Implement and Optimize Autocomplete with Google Places API

Gaël — Fri, 27 Nov 2020 11:35:26 +0000

The Google Places API is a service that returns places predictions using HTTP requests.
The service can be used to provide an autocomplete functionality for text-based geographic searches by returning places such as businesses, addresses and points of interest as a user types.
Google provides a Places library through the Maps Javascript API. This blog post focus on the usage of this library.

Implementing Google Places Autocomplete

Enable the Places API

Before using the Places library in the Maps JavaScript API, you must enable the Places API in the Google Cloud Platform Console.

Go to the Google Cloud Platform Console .
Either create a new or select an existing project.
At the top of the page, click on the button ENABLE APIS AND SERVICES.
Search for Places API, then select it from the results list.
Select ENABLE. When the process finishes, Places API appears in the list of APIs on the Dashboard.

Don't forget to apply Application restrictions (to limit usage for the dedicated APIs - at least Places API) and API restrictions (to enable the use of this API through specific domain). (how to do it)

Loading the library

In order to use autocomplete, you must first load the Google Places library using the libraries parameter in the bootstrap URL for the Google Maps JavaScript API.

<script type="text/javascript" src="https://maps.googleapis.com/maps/api/js?key=YOUR_API_KEY&libraries=places"></script>

Autocompletion using the Widgets

The Places API offers two types of autocomplete widgets, which you can add via the Autocomplete and SearchBox classes respectively.

The `Autocomplete` Widget

autocomplete = new google.maps.places.Autocomplete(input, options);

Autocomplete adds a text input field on your web page. As the user enters text, autocomplete returns place predictions in the form of a dropdown pick list. When the user selects a place from the list, place details is returned in response to a getPlace() request. Each entry in the pick list corresponds to a single place (as defined by the Places API).
You can restrict the search to a particular country and particular place types, as well as setting the bounds.

In the below sample, search for "Old Coff" and select the first item in the pick list (Old Coffee House).
Results are biased towards with the current map bounds using autocomplete.bindTo('bounds', map) but to instruct the Place Autocomplete service to return only results within that region, you should set the strictbounds: true option.

The `SearchBox` Widget

searchBox = new google.maps.places.SearchBox(input, options);

SearchBox adds a text input field to your web page, in much the same way as Autocomplete. The main difference lies in the results that appear in the pick list. SearchBox supplies an extended list of predictions, which can include places (as defined by the Places API) plus suggested search terms.
SearchBox offers fewer options than Autocomplete for restricting the search. You can only bias the search towards a given LatLngBounds.

In the below sample, search for "Coffee near Lond" and select the first suggested search term in the pick list.

Autocompletion using the `AutocompleteService`

service = new google.maps.places.AutocompleteService();
service.getQueryPredictions({ input: 'hotels near Lond' }, function(predictions, status){});

You can create an AutocompleteService object to retrieve predictions programmatically. Call getPlacePredictions() to retrieve matching places, or call getQueryPredictions() to retrieve matching places plus suggested search terms.
AutocompleteService does not add any UI controls. Instead, the above methods return an array of prediction objects. Each prediction object contains the text of the prediction, as well as reference information and details of how the result matches the user input.
This is useful if you want more control over the user interface than is offered by the Autocomplete and SearchBox described above.

To get the location and more information about any of the places which are returned (when a user select the place in the pick list for example), you need to send a Place Details request with getDatails(request, callback) specifying the place_id and wanted fields to be returned.

In the below sample, search for "Old Coff" and select the first item in the pick list (Old Coffee House).

Optimization

Here are different solutions to help you reduce the cost of your Places license.

Return data for specific fields in Place Details requests

When the Places Autocomplete service returns results from a search, it places them within a predictions array.
Each prediction result contains the following fields (check here for more details):
description, place_id, terms, types, matched_substrings, structured_formatting

By default, when a user selects a place from the pick list, autocomplete returns all of the available data fields for the selected place, and you will be billed accordingly.
You can customize Place Detail requests to return data for specific fields used in your application and so decrease the cost of your Places API License. These fields are divided into categories: Basic, Contact, and Atmosphere.

Basic

The Basic category includes fields to locate the place such as the address components and geometry location. see basic fields.

Contact

The Contact category includes the opening hours and fields to contact the business such as the phone number and website. see contact fields.

Atmosphere

The Atmosphere category includes more specific fields (price_level, rating, review,...). see atmosphere fields.

How to specify desired data?

For example, to retrieve only geometry field in the details response (when user select an item in the pick list), define as below:

Places Autocomplete widget: use autocomplete.setFields(['geometry']). check the jsFiddle above to see how.
Places SearchBox widget: there is no way to constrain SearchBox requests to only return specific fields.
Places AutocompleteService: you have to use the PlacesService to call the getDetails(resuest, callback) and specify the fields in your request parameter.

let request = {
  placeId: place_id,
  fields: ['geometry']
}

Per request vs Session Token

If your using the Autocomplete Widget you don't need to implement sessions, as the widget handles sessions automatically in the background.

The Per Request is the default billing option when you use AutocompleteService. Charges are applied per keystroke, that could lead to higher billings.

You should use Session Tokens when implementing AutocompleteService.getPlacePredictions() to group together autocomplete requests for billing purposes. Session tokens group the query and selection phases of a user autocomplete search into a discrete session for billing purposes. The session begins when the user starts typing a query, and concludes when they select a place. Each session can have multiple queries, followed by one place selection.

The following example shows a request using the sessiontoken parameter:

// Create a new session token.
let sessionToken = new google.maps.places.AutocompleteSessionToken();
// Pass the token to the autocomplete service.
let autocompleteService = new google.maps.places.AutocompleteService();
autocompleteService.getPlacePredictions({
  input: 'Coffee near Lond',
  sessionToken: sessionToken
}, displaySuggestions);

Use the Geocoding API

If your application handles user-typed addresses, the addresses are sometimes ambiguous (incomplete, misspelled, or poorly formatted). You can disambiguate addresses using Autocomplete. Then, use the place IDs to get the place locations.
If you have an exact address (or close to it), however, you can reduce costs by using Geocoding instead of Autocomplete.
You could for example use the AutocompleteService to get the exact address name and then execute a geocoding request to get the geometry and others desired fields (instead of AutocompleteService.getDetails()).
See Geocoding API best practices.

Use debounce and minChar with AutocompleteService

The debounce function limits the rate at which a function can fire.
It takes at least 2 arguments, the input function to call and a time to wait before call it.
As long as this method continues to be invoked, the function passed as parameter will not be triggered. It will be called only after the debounce function stops being called for an elapsed time (in milliseconds).

function debounce(func, wait, immediate) {
    var timeout;
    return function() {
        var context = this, args = arguments;
        var later = function() {
            timeout = null;
            if (!immediate) func.apply(context, args);
        };
        var callNow = immediate && !timeout;
        clearTimeout(timeout);
        timeout = setTimeout(later, wait);
        if (callNow) func.apply(context, args);
    };
};

You'll pass the debounce function the function to execute and the fire rate limit in milliseconds.
The following example call the AutocompleteService.getPlacePredictions() on input event of the dedicated input text only when two key strokes are separated by at least 150ms.

autocomplete_input.addEventListener('input', debounce(function() {
    let value = this.value;
    autocompleteService.getPlacePredictions({
        input: value
      }, displaySuggestions);
}, 150));

Autocomplete and SearchBox widgets do not implement a debounce method. Thus each keystroke calls AutocompleteService.

In addition to this mechanism, you could combine your call with a minChar parameter to only trigger the Autocomplete when the text input reaches at least X characters.

autocomplete_input.addEventListener('input', debounce(function() {
    let value = this.value;
    if (value.length > 2) {
       //call here autocompleteService.getPlacePredictions() 
    }
}, 150));

Autocomplete on Stores

If your main goal to implement Google Places Autocomplete is to help your users find the right store for them, why not combine an autocompletion on your stores information (name and city for example) before possibly falling back to the Google Places Autocomplete.
If you're a Woosmap customer and already have your stores hosted on our own, you could use the Woosmap Search API to achieve this.

To easily implement such a solution, have a look at Woosmap MultiSearch. This small JavaScript library is designed to return location suggestions through calling several autocomplete services. It enables you to combine stores search (through Woosmap Search API) and Google Places APIs (Places Autocomplete and Places Details). Moreover, the library implements natively debounce and minchars mechanisms.

If your users are more likely to search for postal codes or localities, the MultiSearch can also retrieve suggestions from Woosmap Localities.

How MultiSearch combines autocomplete services?

Autocomplete services are requested in your desired ordered.
By comparing the user input to the returned results and computing a string matching score between these two values, the library can automatically switch to the next autocomplete service and thereby provide suggestions that better suits the needs.

For more details, check fallback concept doc.

Fallback Example with MultiSearch

The following sample autocomplete for Starbucks coffee shops. If no results are available (insufficient matching score between the field name of Starbucks coffee shops and the user input), a Google places Autocomplete is performed.

In the below sample, search for "London" and select one of the coffee shops suggestion in the pick list.
If you search for "Coleman Coffee", the autocomplete fall back on Google Places (there is no Starbucks coffee with this name).

Conclusion

To optimize Google Places Autocomplete, we recommend the following guidelines:

Prefer using the AutocompleteService instead of the Widgets to gain fine-grained control over the available optimizations.
Use the Per Session billing option.
Gather only required fields when getting details of a place.
Implement Debounce and minChars mechanisms.
Use the Geocoding API to retrieve address information instead of getDetails() from Autocomplete.
Alternatively, offer to autocomplete on your stores information if the first use case is to help users find them.

The last example implements Woosmap MultiSearch with Google AutocompleteService, per Session billing, Debounce, and autocomplete on Stores before fall back to Google. It could be a good entry-point for dealing with Google Places Autocomplete.

Bulk Geocode addresses using Google Maps and GeoPy

Gaël — Wed, 29 Jan 2020 16:23:49 +0000

Geocoding is the process of converting addresses (like a street address) into geographic coordinates (like latitude and longitude).
With Woosmap you can request nearby location or display on a map a lot of geographic elements like stores or any other point of interest. To take advantages of these features, you first have to push geocoded locations to our system.
Most of the time, your dataset has addresses but no location information.

The following script, hosted on our Woosmap Github Organization, is a basic utility for geocoding CSV files that have address data included.
It will parse the file and add coordinate information as well as some metadata on geocoded results like the location type, as discussed below.
It calls Google Maps Geocoding API through the GeoPy python client.

The source code is open to view and change, and contributions are welcomed.

GeoPy

GeoPy is a Python client for several popular geocoding web services.
It makes it easy for Python developers to locate the coordinates of addresses, cities, countries, and landmarks across the globe using third-party geocoders and other data sources.
It includes geocoder classes for the OpenStreetMap Nominatim, ESRI ArcGIS, Google Geocoding API, Baidu Maps, Bing Maps, Yandex, GeoNames, OpenMapQuest and many other geocoding services. The full list is available on the geopy documentation.

Google Maps Geocoding API

Google Maps Geocoding API is a service that provides geocoding and reverse geocoding of addresses.
In this python script we used the GeoPy wrapper of this API but a nice alternative could be to implement the Python Client for Google Maps Services available on Google GitHub organization. Adapting the source code of the script to this python client would be easy as the Geocoding method accepts identical parameters and returns almost the same object.

Registering a Google Maps API key

Each Google Maps Web Service request requires an API key that is available with a Google Account at Google Developers Console.

Steps To get an API key:

Visit the Google Maps Platform page and click Get started.
Select the Maps product to get the APIs that are needed to work with Geopy.
Click Continue.
The Select a project step asks you to associate a name with your use of Google's APIs. Either create a new name or select an existing project.
After agreeing to the terms of service, click Next.
Create a billing account with the Google Maps Platform. A billing account is a requirement in the new Google Maps Platform. For more information, see the Google Maps Platform pricing and billing documentation.

This Key is a long string of generated characters and looks like:

AIzaSyAQanhCC6g4sR3tgj9tmFlByqhGFVKBFZ

Important: This key should be kept secret on your server (but you can revoke a key and generate a new one if needed)

Restrict your API Key

Google Maps API key restrictions are settings you apply to an API key that limit which applications, APIs, and SDKs can be used with that key.
For example, you can specify that an API key can only be used to the Geocoding API from a server with an IP address that matches the server your backend service is running on.

Restricting an API key is fast and easy. You can do it at any time from the credentials page of the Google Cloud console.

Steps:

Go to the Google API credentials page.
Select your project from the menu.
Select the API key that you want to set a restriction on. The API key property page appears.
Under Application restrictions, click IP Address (web servers, cron jobs, ...) and enter the public IP where you're executing the script.
Click Save.

Script Usage

The script takes an input csv file with addresses you need to geocode and produce an output csv file that contains all values from origin csv with appended following fields :

Latitude
Longitude
Location_Type (see below)
Formatted_Address
Error (if needded, for failed geocoded addresses)

Download the script to your local machine. Then run:

python google_batch_geocoder.py

To make it compatible with your own CSV file and Google keys, you have to set the following parameters on top of the script.

Mandatory Parameters

ADDRESS_COLUMNS_NAME - List - used to set a google geocoding query by merging these values into one string comma separated. it depends on your CSV input file
NEW_COLUMNS_NAME - List - appended columns name to processed data csv (no need to change this but you can add new columns depending on [Geocoding Google Results][geocoding_results])
DELIMITER - String - delimiter for your input csv file
INPUT_CSV_FILE - String - path and name for input csv file
OUTPUT_CSV_FILE - String - path and name for output csv file

Optional Parameters

COMPONENTS_RESTRICTIONS_COLUMNS_NAME - Dict - used to define component restrictions for google geocoding. See [Google componentRestrictions doc][component_restrictions] for details.
GOOGLE_API_KEY - String - Google API Key set above.

Input Data

The sample data (hairdresser_sample_addresses.csv) supported by default is a CSV file representing various hairdressers around the world. You can see below a subset of what the file looks like:

name	addressline1	town	postalcode	isocode
Hairhouse Warehouse	Riverlink Shopping centre	Ipswich	4305	AU
Room For Hair	20 Jull Street	ARMADALE	6112	AU
D'luxe Hair & Beauty	Burra Place Shellharbour City Plaza	Shellharbour	2529	AU

Script Explanation

Prepare the Destination File

As described above, the destination file contains all the origins fields plus geocoded data (latitude, longitude and error when occur) and some metadata (formatted_address and location_type). In the case of our sample file, the Header of destination CSV file looks like :

name;addressline1;town;IsoCode;Lat;Long;Error;formatted_address;location_type

To build it, open the input and output csv files and create a new header to append to destination file:

with open(INPUT_CSV_FILE, 'r') as csvinput:
    with open(OUTPUT_CSV_FILE, 'w') as csvoutput:
        # new csv based on same dialect as input csv
        writer = csv.writer(csvoutput, dialect="ga")

        # create a proper header with stripped fieldnames for new CSV
        header = [h.strip() for h in csvinput.next().split(DELIMITER)]

        # read Input CSV as Dict of Dict
        reader = csv.DictReader(csvinput, dialect="ga", fieldnames=header)

        # 2-dimensional data variable used to write the new CSV
        processed_data = []

        # append new columns, to receive geocoded information 
        # to the header of the new CSV
        header = list(reader.fieldnames)
        for column_name in NEW_COLUMNS_NAME:
            header.append(column_name.strip())
        processed_data.append(header)

Build an address line

The principle is to build for each row a line address by merging multiple values in one string, based on the list ADDRESS_COLUMNS_NAME, to pass it to Google Geocoder. For instance, the first line of our csv will become the line address "Hairhouse Warehouse, Riverlink Shopping centre, Ipswich".

# iterate through each row of input CSV
for record in reader:
    # build a line address 
    # based on the merge of multiple field values to pass to Google Geocoder`
    line_address = ','.join(
        str(val) for val in (record[column_name] for column_name in ADDRESS_COLUMNS_NAME))

Apply the Component Restrictions

Google Maps Geocoding API is able to return limited address results in a specific area. This restriction is specified in the script using the filters dict COMPONENTS_RESTRICTIONS_COLUMNS_NAME.
A filter consists in a list of pairs component:value. You can leave it empty {} if you don't want to apply restricted area.

# if you want to use componentRestrictions feature,
# build a matching dict {'googleComponentRestrictionField' : 'yourCSVFieldValue'}
# to pass to Google Geocoder
component_restrictions = {}
if COMPONENT_RESTRICTIONS_COLUMNS_NAME:
    for key, value in COMPONENT_RESTRICTIONS_COLUMNS_NAME.items():
        component_restrictions[key] = record[value]

Geocode the address

Before call the geocoding method of GeoPy, instantiate a new GoogleV3 Geocoder with your Google credentials, at least the Server API Key.

geo_locator = GoogleV3(api_key=GOOGLE_API_KEY,
                       client_id=GOOGLE_CLIENT_ID,
                       secret_key=GOOGLE_SECRET_KEY)

Then call the geocoding method passing the geocoder instance, built line address and optional component restrictions.

# geocode the built line_address and passing optional componentRestrictions
location = geocode_address(geo_locator, line_address, component_restrictions)

def geocode_address(geo_locator, line_address, component_restrictions=None, retry_counter=0):
    # the geopy GoogleV3 geocoding call
    location = geo_locator.geocode(line_address, components=component_restrictions)

    # build a dict to append to output CSV
    if location is not None:
        location_result = {"Lat": location.latitude, "Long": location.longitude, "Error": "",
                           "formatted_address": location.raw['formatted_address'],
                           "location_type": location.raw['geometry']['location_type']}
    return location_result

Retry on Failure

The script retries to geocode the given line address when intermittent failures occur.
That is, when the GeoPy library raised a GeocodeError exception that means that any of the retriable 5xx errors are returned from the API.
By default the retry counter (RETRY_COUNTER_CONST) is set to 5:

# To retry because intermittent failures sometimes occurs
except (GeocoderQueryError) as error:
    if retry_counter < RETRY_COUNTER_CONST:
        return geocode_address(geo_locator, line_address, component_restrictions, retry_counter + 1)
    else:
        location_result = {"Lat": 0, "Long": 0, "Error": error.message, "formatted_address": "",
                           "location_type": ""}

Handle Generic and Geocoding Exceptions

Other exceptions can occur, like when you exceed your daily quota limit or request by seconds.
To support them import the GeoPy exceptions and handle each errors after geocode call.
The script also raises an error when no geocoded address is found.
The error message is appended to Error CSV field.

# import Exceptions from GeoPy
from geopy.exc import (
    GeocoderQueryError,
    GeocoderQuotaExceeded,
    ConfigurationError,
    GeocoderParseError,
)

# after geocode call, if no result found, raise a ValueError
if location is None:
    raise ValueError("None location found, please verify your address line")

# To catch generic and geocoder errors.
except (ValueError, GeocoderQuotaExceeded, ConfigurationError, GeocoderParseError) as error:
    location_result = {"Lat": 0, "Long": 0, "Error": error.message, "formatted_address": "", "location_type": ""}

Query-per-Second

You can exceed the Google Maps Platform web services usage limits by sending too many requests per second. In that case, you will raise an OVER_QUERY_LIMIT error. You will have to wait between two API Call.

# for too many requests per second, we have to sleep 500 ms between each request.
if not GOOGLE_SECRET_KEY:
    time.sleep(0.5)

Results

Apart from the Latitude and Longitude fields, here are the 3 main results appended to destination CSV file.

Formatted Address

The formatted_address matches a readable address of the place (and original line address). This address is most of the time equivalent to the "postal address".

Geocoding Accuracy Labels

For each succeeded geocoded address, a geocoding accuracy results is returned in location_type field. It comes from the Google Maps API Geocoding service (see Google geocoding Results doc for more information). The following values are currently supported:

"ROOFTOP" indicates that the returned result is a precise geocode for which we have location information accuracy down to street address precision.
"RANGE_INTERPOLATED" indicates that the returned result reflects an approximation (usually on a road) interpolated between two precise points (such as intersections). Interpolated results are generally returned when rooftop geocodes are unavailable for a street address.
"GEOMETRIC_CENTER" indicates that the returned result is the geometric center of a result such as a polyline (for example, a street) or polygon (region).
"APPROXIMATE" indicates that the returned result is approximate.

Error

Whenever the call to Google Maps API failed, the script will return an error message "None location found, please verify your address line".
For all other errors, the message raised by GeoPy will be appended to the field value. See GeoPy Exceptions for more details.

Display your POI on a Map

You could use Woosmap to display the geocoded addresses on a Map. If so, follow this help article to upload your CSV file to Woosmap.

Here is a sample of what it looks like on the Woosmap Store Locator widget.

Useful Links

This python script on the Woosmap Github Organization
GeoPy
Alternative to GeoPy : Python Client for Google Maps Services
Google Geocoding Strategies
Google geocoding Results doc
More info on Woosmap.