DEV Community: adrienshen

Creating a Data Pipeline from Airtable to Backend

adrienshen — Thu, 16 Jan 2020 15:39:04 +0000

I often work with e-commerce systems and we frequently need to get product data into whatever e-commerce system we are using whether that might be Saleor, Magento, Wordpress, Shopify, or whatever comes next. One tool we find useful is Airtable for administration staff to fill in all the product info first in an organize manner appropriate for their team. Then afterwards, we could write a script in Python to import the rows through the Airtable API into our application database. The Airtable API is straightforward to use, but there are some considerations and issues we came across. I will try to outline below to make it easier for other developers.

Some technologies and tools we will be covering in this article:

Airtable and Airtable API
Python and Django (Saleor ecommerce framework)
GraphQL queries and mutations

For those not familiar with Airtable, it’s a user friendly database spreadsheet tool that has a nicely documented API that’s customized to each base you have. It has a generous free tier and I find it useful for various tasks from lightweight CMS systems to offering easy configuration settings to clients.

GraphQL is REST alternative and allows developers to query for exactly the data they need. One of the beat points of GraphQL is that it is self documenting and typed query language. Example of GraphQL query:

import gql from 'graphql-tag';
import { useQuery } from '@apollo/react-hooks';

const GET_DOGS = gql`
  {
    dogs {
      id
      breed
    }
  }
`;

Preparing Airtable columns and API Key

Airtable is really intuitive and simple to use. That’s why it’s a good tool for non-technical folks like office admins and data entry. As an example we have a product base like that contains the columns:

sku: Text
name: Text
description: Long Text
base_price: Number
publication_date: Date
is_published: Boolean
weight: Decimal
quantity_opt: Number
images_opt: Attachments (will explain more later)
category: SingleSelect
product_pk_auto: Text (ID)
has_been_imported: Boolean

There are some optional parameters for the GraphQL API that we’ll be posting the mutation to, so we can mark that with _opt suffix to let the administrators know the field is optional.

After all the columns and some example data is setup in the Airtable base, AirTable will provide us with a custom API filled in with our columns and example data.

We need to grab the BASE_ID and API_KEY to use in our GET request. Our API_KEY is on the left panel, just check “show API key”. For the purpose of importing a list of products to our database, we will need to use the “List collection records” endpoint. The example curl requests looks like this:

    curl "https://api.airtable.com/v0/appC7p39pnmKYOJ17/Collections?maxRecords=3&view=Grid%20view" \
      -H "Authorization: Bearer YOUR_API_KEY"

The general import logic that we want to is:

Grab the list of all rows from Airtable
Get authentication token from our e-commerce API if there is one
Make sure to get any mappings like category → category_ids from our system
Map each row from Airtable into our systems request object
Create the POST request to update our system (GraphQL mutation in my case)
Get the product_id and update Airtable with it
Download and process any images in our application
Repeat with each row until all rows are exhausted

In our case, we like to implement a custom Django command to initiate the imports task. It’s quite nice and possible later to run on a schedule cron job if we desire. Here is the handler function of the script to get the list of all rows from Airtable, authentication, sending mutation, and updating back to Airtable. In subsequent steps, we will fill in the sub function logic.

    from django.core.management.base import BaseCommand, CommandError
    import requests
    from pprint import pprint
    import json
    import os

    API_KEY = 'airtable-api-key';

    class Command(BaseCommand):

      def handle(self, *args, **options):
        # pull records from airtable
        response = requests.get('https://api.airtable.com/v0/appC7p39pnmKYOJ17/Products?maxRecords=1000&view=Grid%20view', headers={'Authorization': 'Bearer ' + API_KEY})

        product_rows = response.json().get('records')
        pprint('product rows >> ', product_rows)

        json = self.get_permission_token()
        if len(json['data']['tokenCreate']['errors']) is not 0:
          print('Abort importing, Authentication failed.')
          return
        print('New token created: {} for import user: {} \n\n'.format(json['data']['tokenCreate']['token'], json['data']['tokenCreate']['user']['email']))
        headers = {
          'Authorization': 'JWT {}'.format(json['data']['tokenCreate']['token'])
        }

        category_mappings = self.get_categories();

        for pr in product_rows:
          fields = pr.get('fields', None)

          if not fields or fields.get('has_been_imported') or fields.get('product_id_pk'):
            print('No fields or row has already been imported: sku={}, name={}'.format(fields.get('sku'), fields.get('name')))
            continue
          create_input = self.prepare_input(fields=fields, collections_mappings=collections_mappings)

          results = self.send_mutation(create_input, headers)
          if not results:
            self.handle_error()

          self.update_airtable(pr['id'], results)

      def prepare_input():
        pass

      def get_permission_token():
        pass

      def get_categories(self):
        pass

      def send_mutation(self):
        pass

      def update_airtable(self):
        pass

      def handle_error(self):
        pass

In the next sections we will implement each stage of the logic by filling in the functions. We are going to have a main for loop that goes through each record, so if we have an authentication token for our app which mostly likely you will, we need to get this token before entering into the loop. An example that works for our case is shown below:

 def get_permission_token():
   mutation = '''
     mutation TokenCreateMutation($email: String!, $password: String!) {
       tokenCreate(email: $email, password: $password) {
         token
         errors{
           field
           message
         }
         user{
           id
           email
           }
         }
       }
   '''
   input = {"email": auth_email, "password": auth_password}
   response = requests.post('http://localhost:8000/graphql/', json={'query': mutation, "variables": input})

Now that we have the auth token, we can use it every time we need to update anything on our e-commerce system. The next step would need to make sure the data is mapped correctly to the format our API will accept. We also have to make sure that any columns that are not optional needs to be provided and have the record skipped with optional error handling if required fields are missing. Every API and system is different, so you will have to adjust it so it works for you. I will show my implementation below for Saleor ProductCreate API:

def prepare_input(self, fields):
  if (fields.get('name') and fields.get('base_price') and fields.get('category') and fields.get('description') and fields.get('publication_date') and fields.get('weight')):
    input = {
      'publicationDate': fields.get('publication_date'),
      'name': fields.get('name'),
      'description': fields.get('description'),
      'isPublished': fields.get('is_published'),
      'weight': fields.get('weight'),
      'basePrice': fields.get('base_price'),
      'sku': fields.get('sku'),
      'weight': fields.get('weight'),
      # just pass the first image
      'airtableImageUrl': fields.get('images_opt')[0]['url']
    }
 # handle mappings for category

We also need to make sure relational mappings are correct during the imports process. So Product: wall cabinet A might belong to the Category: “Kitchen Sets”. In APIs this is usually done through passing a list of IDs like {'category': 'Q2F0ZWdvcnk6MTA'} to the create mutation. On the Airtable interface, we want the ids to be human readable inputs like: Kitchen Sets. We can keep the mappings as a hard-coded dictionary in the imports scripts or we can dynamically get the list of products from our application.

 def get_categories(self):
     query = '''
       query categories($first: Int) {
           categories(first: $first) {
               edges {
                   node {
                       id
                       name
                   }
                   cursor
               }
               totalCount
           }
       }
   '''
   response = requests.post(GRAPHQL_BASE, json={'query': query, 'variables': { 'first': 20 }})
   json = response.json()
   category_mappings_to_id = {}
   for category in json['data']['categories']['edges']:
     category_mappings_to_id[category['node']['name']] = category['node']['id']
   return category_mappings_to_id

   def prepare_input(self, fields, category_mappings):
     # basic input fields...
     # handle mappings for category and any other related models
     if fields.get('category'):
       input['category'] = category_mappings[fields.get('category')]

Now that the input is how we want it, we need to actually send the mutation through to our API to update the rows in the database. We will implement the send_mutation function now. We handle the error case if results from API is empty and return the product to the handler function.

 def send_mutation(self, input, headers):
     mutation = '''
       mutation ProductCreateMutation($input: ProductCreateInput!) {
           productCreate(input: $input) {
               errors {
                   field
                   message
               }
               product {
                   id
                   name
                   description
               }
           }
       }
   '''

   response = requests.post(GRAPHQL_BASE, json={'query': mutation, "variables": { "input": input }}, headers=headers)
   json = response.json()
   print('Product updated :: ', json['data']['productCreate'])
   if len(json['data']['productCreate']['errors']) is not 0:
     print('Import of {} failed'.format(input['name']))
     return None
   else:
     return json['data']['productCreate']['product']

Once, we are sure that the product has been updated and there were no errors in our application, we will want to update the Airtable row to reflect that the product now exists in our application database. This also ensures that the same script can be run multiple times as more products are added to the Airtable sheet as we can then skip the records that have already been imported.

 def update_airtable(self, id, updated):
   patch_payload = {
     "records": [{
       "id": id,
       "fields": {
         "has_been_imported_auto": True,
         "product_id_pk_auto": updated['id'],
       }
     }]
   }

   response = requests.patch('https://api.airtable.com/v0/<replace-with-your-baseid>/Products', json=patch_payload, headers={'Authorization': 'Bearer ' + api_key})
   data = response.json()
   print('Airtable id updated: {} \n'.format(data['records'][0]['id']))

So one way to do this is to have the application model save the airtable_image in another column, then use a background job to download the image attachment from the url saved. The specifics of background jobs are outside the scope of this tutorial, but the image download logic might look something like this:

 # background tasks file in Django
import requests
import tempfile

 @app.task
 def save_airtable_product_image(pk):
   instance = Product.objects.get(pk=pk)
   temp = download_airtable_attachment_image(instance.airtable_image_url)
   file_name = instance.airtable_image_url.split('/')[-1]
   pi = ProductImage()
   pi.product = instance
   pi.image.save(file_name, files.File(temp))
   create_product_thumbnails(pi.pk)

 def download_airtable_attachment_image(image_url):
   '''Download airtable (https://dl.airtable.com/.attachments) image and save to filesystem, then save reference in product_collection.background_image, then generate thumbnails for various sizes
   '''
   request.get(image_url, stream=True)
   if request.status_code != requests.codes.ok:
     print('Could not download image {}'.format(image_url))

   temp = tempfile.NamedTemporaryFile()
   for block in request.iter_content(1024 * 8):
     if not block:
       break
     temp.write(block)

   return temp

That sums up the main tutorial on creating a pipeline from Airtable to your E-commerce application. Some further improvements to try might be to add Slack alerts for team members with import summary or to automate the task on a daily schedule. Will leave this to the reader to try.

I enjoy building e-commerce applications on a variety of technology stacks including Django, Node.js, and React. Even more enjoyable is helping merchants succeed and and scale with technology. I hope my writing benefits developers and if you know merchants who could use digital technologies to enhance their business,feel free to reach out to to chat!

Hacking together your own Youtube Suggest API

adrienshen — Mon, 13 Jan 2020 12:02:18 +0000

Premise

I had some extra time last weekend and decided to work on a small project that would solve my own problem. It’s suppose to be a really simple mobile PWA that only streams Youtube audio and loops repeatedly. There’s a bunch of them out there so it’s not exactly an original idea, but I wanted something simple without all the extra cruft like ads, comments, or authentication.

Feature to Implement

Right off the bat, there was a core functionality I needed to implement. A “search box” for querying the YT api for videos. The API for video search is straightforward enough: https://developers.google.com/youtube/v3/docs/search/list. Just sign up for the Youtube Data API key and we have access to all the power of Youtube search.

Now, we could just build one that the user enters in the query and then taps a button to search, but it would be much more user friendly if there was autocomplete suggest like the actual Youtube site. Auto suggest would make it so much easier to use. Here is an image of desire functionality from youtube.

No API 😞

But, alas! After a couple hours of searching on both Google and Youtube API docs, it turns out Google does not provide an official auto-suggest API. Or it seems, they provided the API previously, but decided to shut it down for some reason. Now at this point, I could just go ahead with the core search functionality and forget about the auto suggestions… but let’s have a look at Youtube first just out of curiosity.

Looking underneath the hood

Over at Youtube, if we start typing in the search bar, and open up the Chrome dev tools, we see network requests being made that points to an undocumented API endpoint: https://clients1.google.com/complete/search?client=youtube&hl=en&gl=sg&gs_rn=64&gs_ri=youtube&tok=h3yTGb1h3-yuCBwsAaQpxQ&ds=yt&cp=3&gs_id=2u&q=jaz&callback=google.sbox.p50&gs_gbg=0l0MjG05RWnWBe9WcipQbsy

After playing around with the parameters, it turns out that most of the params are not really needed. The main ones that matter to our use case is:

client: forces json response, we want to use youtube here
ds: google site properties, use yt to restrict to Youtube
hl: the culture or language. use for localization. Default is usually en
callback: this is the jsonp callback
q: term query you want to search for

At this point, the endpoint works. If you tried it in the browser now, you would download a text. It’s strange text file with numbers and gibberish, but inside it we clearly see the data we need to implement our autocomplete search. Hooray!



// contents of autosuggest endpoint
google.sbox.p50 && google.sbox.p50(["jazz ",[["jazz music",0],["jazz piano",0],["jazz songs",0],["jazz dance",0,[131]],["jazz music best songs",0],["jazz instrumental",0],["jazz guitar",0],["jazz relaxing music",0,[131]],["jazz jennings",0],["jazz for work",0,[131]]],{"a":"FCwlE6frPjfCHAJSPzskH5xxMxJia3UhfNxNRVG6aehsz7iBn4XxJQ6ACUGMVuaAl5f1LHrO2ErGn7t4d6mIXg965Zxp3bENM4iS00nEvwhiiSe8Bi39NZsbdj2BHz3FD0C","j":"32","k":1,"q":"8KKe7s-xREtd_veunmBB7oKGghg"}])

You might recognized this as jsonp and if not, then a few google searches, and we have the answer! The google.sbox.p50 is the callback function which we’ll pass in ourselves.

In my side project, I’m using axios, and we can find a jsonp adaptor for axios here. The basic request logic looks like this:



    export const suggest = (term: string) => {
      const GOOGLE_AC_URL: string = `https://clients1.google.com/complete/search`;
      return axios({
        // A YT undocumented API for auto suggest search queries
        url: GOOGLE_AC_URL,
        adapter: jsonpAdapter,
        params: {
          client: "youtube",
          hl: "en",
          ds: "yt",
          q: term,
        }
      })
      .then((res: AxiosResponse) => {
        console.log("jsonp results >> ", res);
        if (res.status !== 200) {
          throw Error("Suggest API not 200!");
        }
        return res.data[1].map((item: any[]) => item[0]);
      })
    }

Now, we just need to connect the results to a react component input element and wire up the state and change handlers. I have used styled-components components to rename and style many of the html elements. The bare bones implementation is show here:




    /* search-box component */
    import * as React from "react";
    import styled from "styled-components";
    import * as _ from "lodash";

    import {youtubeSearch, suggest} from "./services/youtube";

    export default class SearchBox extends React.PureComponent<any, any> {
      public state = {
        suggestions: null,
      };

      constructor(props: any) {
        super(props);
        this.onTypeSuggest = _.debounce(this.onTypeSuggest, 500, { leading: true });
      }

      public render() {
        return (
          <>
            <Container>
              <SearchBox
                onChange={e => this.onTypeSuggest(e.target.value)}
                placeholder="Search for music, songs, podcasts"
                type="search"
              />
            </Container>
            <Suggestions onSearch={this.onSearch} items={this.state.suggestions} />
          </>
        )
      }

      private onTypeSuggest = async (
        queryString: string,
      ) => {
        if (queryString.length < 5) {
          // search only after 5 chars
          return null;
        }
        const list = await suggest(queryString);
        return this.setState({ suggestions: list });
      }

      // Search the term when selected using the official youtube search api:
      // https://www.googleapis.com/youtube/v3/search
      private onSearch = async (queryString: string, platform: MediaPlatforms) => {
        if (platform === MediaPlatforms.Youtube) {
          const platformSearchResults = await youtubeSearch(queryString);
          this.setState({ suggestions: null });
          this.props.update(platformSearchResults);
        }
      }
    }

We also want to show the suggestions that we get from our hand crafted youtube autosuggest api. A simple list of items will do. Each suggestion is passed the onSearch function which takes the autosuggestion selected and queries the official youtube search api above.



function Suggestions({ onSearch, items }: any) {
  if (!items || !items.length) {
    return null;
  }
  return (
    <Section>
      {items.map((item: string, key: number) => {
        return <SuggestionRow onSearch={onSearch} key={key} text={item} />;
      })}
    </Section>
  );
}

function SuggestionRow({ onSearch, text }: any) {
  return (
    <SuggestionSpan onClick={() => onSearch(text, MediaPlatforms.Youtube)}>
      {text}
    </SuggestionSpan>
  );
}

And add some styling to pretty things up a bit.



const Container = styled("div")`
  position: relative;
  width: 100%;
  display: flex;
  flex-flow: row nowrap;
  align-items: center;
  justify-content: flex-start;
`;
const SearchBox = styled("input")`
  box-sizing: border-box;
  height: 2.9rem;
  width: 100%;
  padding: 0.5rem;
  padding-left: 1rem;
  border-radius: 0.2rem;
  border: 2px solid #aaa;
  max-width: 800px;
  font-size: 1rem;
`;
const CSearchIcon = styled(SearchIcon)`
  cursor: pointer;
  margin-left: -50px;
`;
const Section = styled("section")`
  width: 95%;
  min-height: 18rem;
  height: auto;
  border: 1px solid #ddd;
  border-top: none;
  border-radius: 5px;
  margin-top: 1rem;
  padding: 0.5rem;
  box-shadow: 1px 1px 1px #ddd;
  z-index: 1000;
`;
const SuggestionSpan = styled("span")`
  display: inline-block;
  width: 100%;
  color: #9c27b0;
  font-weight: 800;
  margin-bottom: 0.5rem;
  margin-left: 0.5rem;
  cursor: pointer;
  z-index: 1000;
`;

Awesome. Looks like it works as expected!

There might be many creative uses of the autosuggest api, not just for the obvious search suggestions. Would be interesting to hear some others. But make sure to use responsibly!

Finishing up and next steps

Now that the basic functionality works. We could package it up nicely into a react hook so it’s easily usable in any component or by anyone with an npm install

If you want to reverse engineer the Youtube stream, that’s an another level of complexity and very interesting topic to explore. A good starting point would be here: https://tyrrrz.me/blog/reverse-engineering-youtube and checkout out the node-ytdl source or just use the package!

I may write follow-up post on Node.js audio streaming once I learn more about the topic. Also, I would like to write about the intricacies of audio playback for mobile browsers such as Chrome and Safari as it relates to PWAs in a future post.

Cheers.

IndexedDB with Push Notifications

adrienshen — Mon, 25 Jun 2018 05:31:26 +0000

Introduction to Progressive Web Apps

Ever since Jake Archibald reveal a glimpse of what progressive web apps and service workers can do in the Google I/O 2016 event, I have been fairly optimistic about the potential applications this new set of web technologies can solve in developing the next generation of mobile web applications. PWAs allowed the discoverability of the open web to merge with the engagement and responsiveness of the native apps. It is really the best of both worlds. PWAs do this with rigorously caching website assets and scripts via service workers, using service workers for push notifications and background sync, and features like "Add to Home Screen" to give the native app like feel. This is just a simple introduction of Progressive Web Apps. To learn more, just out the full Google PWA tutorials and the various Google I/O talks. What I want to show in this post is the usages of accessing IndexedDB on receiving a web push notification and how it can be used to implement some pretty neat features.

Push Notifications on the Web

One of the core features that the Service Worker makes possible is to allow the web browser to receive web push notifications which used to be in the exclusive domain of native mobile applications. This feature is implemented in every major desktop and mobile browser now except Safari. Businesses, for example, can use these web push notifications to notify that a customer payment was made to a merchant progressive web app used by the cashier side. There is a server-side implementation involve that needs to send the push to the designated push service, but you can use curl or Postmen to simulate that. The client application push listener code in the service worker file will look something like this:

// most basic push notification example
self.addEventListener('push', function(event) {
    if (event.data) {
        const pushTitle = "A simple push notification";
        const pushOptions = {
            body: "push content body",
            icon: "./images/logo-192x192.png",
            vibrate: [100,50,100],
        };

        self.registration.showNotification(
            pushTitle,
            pushOptions,
        )
    }
});

When the device receives a push from the browser specific push service, the Service Worker will wake up for the moment and execute the code inside the handler and display the appropriate push message to the user. There are many settings that can be configured in the push such as priority levels, badges, tags, and event sounds. For a more comprehensive explanation see here: https://web-push-book.gauntface.com/

Scope of Service Workers

Service Workers is an amazing tool and what makes the modern generation of native like web applications possible. There are many introductions to Service Workers such as this one here, but the important concept to realize here is that Service Workers do not run in the same environment as our regular Javascript code. It means that Service Workers do not have access to normal browser APIs like the DOM, nor can it access APIs like Geolocation or LocalStorage. This was not obvious to me at first and can cause some frustrations in the features we need to implement. Instead of directly accessing the page DOM elements, the Service Worker will need to send a message through an API such as PostMessage which the app will listen to, and respond with a handler. But then, what if we need functionalities like user Geolocation at the time of say, a Push Notification event? This is where IndexDB comes in.

Why use IndexedDB?

IndexedDB is the recommended web browsers data object store for large amounts of data and more structure storage. There are other persistent storage on the browser like LocalStorage/SessionStorage, WebSQL, and AppCache as well. LocalStorage has certain limitations such as 5mb to 10mb maximum storage on most browsers and all data is saved in strings. I only use it for simple scenarios such as remembering if a user has already seen the tutorial slides for instance. WebSQL and the old Application Cache are no longer being supported so probably not recommend to be used when developing an offline app in 2018.

IndexedDB has:
- Low, level flexible interface for developers
- Mostly asynchronous
- Transactions, all database operations must be part of a transaction
- Most importantly, it is accessible via Service Workers which is our primary usage here

Store Geolocation in IndexedDB

We know that Service Workers do not permit access to Geolocation API directly nor can it access LocalStorage. So let us store the Geolocation data in IndexDB instead. We will be using Jake's idb helper library so we can utilize Promises which provides a cleaner interface.

// idb.js
import idb from 'idb';

export const idbPromise = idb.open('realtime-app', 2, db => {
    switch (db.oldVersion) {
        case 0:
        // placeholder so that block executes if there's no db
        case 1:
            console.log("creating object store: deviceGeolocation");
            db.createObjectStore("deviceGeolocation", {autoIncrement: true});
    }
});

We need a way to preserve the integrity of the database updates, so we export a helper idbPromise that will take care of database updates for users with different database versions. Next, we use the Geolocation API to query the user device location.

import { idbPromise } from "idb.js"

export const calculateGeoLocation = () => {
    if (navigator.geolocation) {
        navigator.geolocation.watchPosition(position => {
            _storeGeoIndexedDB(position.coords);
        })
    }
}

const _storeGeoIndexedDB = ({Lat, Lon}) => {
    if (!window.indexedDB) return;
    const Timestamp = new Date();

    idbPromise.then(db => {
        let tx = db.transaction('deviceGeolocation', 'readwrite');
        let deviceGeolocation = tx.objectStore('deviceGeolocation');
        let entry = { Lat, Lon, Timestamp };
        deviceGeolocation.add(entry);

        return tx.complete;
    })
}

We create a function that will query the geolocation and store that as an object inside our deviceGeolocation database using the helper idb Promise we created earlier. Now we can use the calculateGeoLocation anywhere in our application that we see fit. We can even run it on a interval or as a response to a user action. In my case, I usually ask for geolocation as part of the app on-boarding process tied to a user driven action.

This is how it looks like in the Chrome Application Tab

Notes on Geolocation API

There are two main native web geolocation APIs available to use for developers. Geolocation.getCurrentLocation() and Geolocation.watchPosition(). We use Geolocation.watchPosition here as the latter supposedly has better accuracy for mobile devices and will update the position when device location changes. Both methods allow passing in a PositionOptions parameter to further customize for your specific needs.

For better accuracy set the PositionOptions.enabledHighAccuracy to true, but this might result in slower response times and drain device battery quicker.

Another useful option is PositionOptions.maximumAge which will be the amount of time in milliseconds the device is allowed to returned a cache geo position. Again, this depends on the specific use case and can save some calculations if the user location does not have to be super up-to-date or accurate.

Contrived Use Case Example

Now we want to combine Service Workers, Push Notification API, IndexedDB, and UserGeolocation together to build some useful feature for our mobile application. Imagine we have completely fictional mobile application that is to predict volcano eruptions in our cloud servers and administers a push notification to user devices when there is potential eruption. The push payload from backend will look like this.

{"data" : {
    "Lat" : 2.18,
    "Lon": 102.89,
    "Vei" : 6.8,
    "EruptionStartTime" : "2018-06-20 10:55:00.764269",
    "Region": ""},
    "type" : "SERIOUS"
}

The Lat and Lon is of the volcanic eruption coordinates. The EruptionStartTime is the future predicted eruption time by our servers. The VEI is the calculated intensity of this eruption.

Let's say the requirement is to have the progressive web app receive the push in the background and then decide the relative impact based on the users current location. In other words, should he run for his life or is it okay to just chill. One thing we need to do is calculate distance from EruptionPoint and user current location. We might also want to calculate how long they have until the heat waves and poisonous gases get to them. Then we want to show notification displaying all the super useful data.

Accessing IndexedDB in Push Event

Let us go back to the push listener event. We need to access our user stored location inside IndexedDB, make all our calculations, and then display to the user a relevant push message all within the push event.

// contrive use case example
self.addEventListener('push', function(event) {
    if (event.data) {
        const pushTitle = "Volcanic Alert";
        const pushOptions;
        const pushEventData = event.data.json();
        const eruptionPayload = JSON.parse(pushEventData.data);

        if (pushEventData.type === "SERIOUS") {
            // Access IndexDB for our geolocation data using the same
            // idb helper from earlier
            let notifResults = idbPromise().then(db => {

                let tx.db.transaction("userGeolocation", "readonly");
                let userGeolocation = tx.objectStore("userGeolocation");
                return userGeolocation.getAll();
            }).then(userGeolocationRecords => {

                const mostRecentGeolocation =
                    userGeolocationRecords[userGeolocationRecords.length - 1];

                // Calculate spherical distance from eruption
                const deviceSphericalDistanceFromEruption =
                    getSphericalDistance(
                        mostRecentGeolocation["Lat"],
                        mostRecentGeolocation["Lon"],
                        eruptionPayload["Lat"],
                        eruptionPayload["Lon"],
                    );

                // Calculate relative intensity
                const intensityReadableString = calculateIntensity(
                    deviceSphericalDistanceFromEruption,
                    eruptionPayload['Vei'],
                )

                // We might also want the relative time for impact to arrive
                const timeTillSeconds = getTimeTill(deviceSphericalDistanceFromEruption, eruptionPayload["EruptionStartTime"]);

                payload.intensityHumanReadable = intensityHumanReadable;
                payload.timeTillSeconds = timeTillSeconds;

                // Now return the calculate data from the promise
                return {
                    body: "Intensity: " + intensityReadableString,
                    tag: "alert",
                    data: payload,
                    renotify: true,
                    requireInteraction: true, // it's pretty important for him to see this
                }
            });

            // Now finish with event.waitUntil()
            event.waitUntil(
                ...
            )
        }

        // Handle other types here...
    }
});

Show Notification in event.waitUntil()

We know that the query to IndexDB is asynchronous and will return us a promise. This promise chain is exactly what event.waitUntil expects. event.waitUntil will wait until a promise is resolved and then show the notification, then terminate the service worker process. This is the trick to having indexedDB and Web Push work together in harmony. This part is super important to understand to avoid a ton of headache surrounding using service workers with asynchronous APIs.

// Finishing up with notification
  event.waitUntil(
    notifResults.then(notifOptions => {
      isClientFocused().then(clientOpts => {
          clientOpts.windowClients.forEach(windowClient => {
            windowClient.postMessage({
              message: 'Received a volcano alert.',
              time: new Date().toString(),
              intensity: notifOptions.data.intensityReadableString,
              seconds: notifOptions.data.timeTillSeconds,
            });
          });

        self.registration.showNotification(
          pushTitle, // Volcanic Alert
          notifOptions,
        )
      })
    })
  )
}

We should also handle the case where the user has the application already opened on his device for whatever reason. In this case, we should post the message from the Service Worker to the client app using an API like PostMessage illustrated above. I use a helper method isClientFocused() here which which will give me open clients if available. If you do not need to handle this case, then you can ignore that part and just show the notification directly.

Further Considerations

Always show a notification. Sometimes you will see Chrome display "This site has been updated in the background." The browser expects a notification to be shown and either a notification is not being shown after event.waitUntil() is finished or something is broken. Just check to see if self.registration.showNotification is being called and no errors are happening before that happens.

Handle the notification click. While not the subject of this post, make sure to consider what happens after the notification click event. Perhaps it takes them to the app url showing them a countdown of the eruption in our case.

Service Workers can be quite tricky. I have had my fair experience of mistakes and surprises over the months using Service Workers in production - sometimes with very high traffic sites. Perhaps I will write about it one day, but I always like to just link to this video: https://www.youtube.com/watch?v=CPP9ew4Co0M. As stated in the video, service workers can be a complex beast, so using a generator like sw-precache can be very helpful.

Making sure to test on Localhost. Service Worker installation requires HTTPs to prevent attacks such as man-in-the-middle, but localhost will work as an exception to that rule.

Communicating with your core app. Since the Service Worker acts as an proxy server separate from the web page, we need to use special methods to communicate between the two separate context. Two methods I have used in the past are PostMessage() and passing data through urls on notificationclick.

Hope the post was helpful, and let me know if you have any questions regarding similar implementations.

sources

https://developer.mozilla.org/en-US/docs/Web/API/Service_Worker_API/Using_Service_Workers
https://developer.mozilla.org/en-US/docs/Web/API/IndexedDB_API/Using_IndexedDB
https://geology.com/stories/13/volcanic-explosivity-index/

How hidden files can screw you over when deploying progressive web apps

adrienshen — Mon, 25 Jun 2018 05:30:08 +0000

Hidden files such as the DS_Store on macOS can really make your life miserable and produce strange abnormalities during deployment if you are not paying attention. Read on to find out how I wasted 1 hour trying to figure out why the service worker on my progressive web app was not working correctly.

Background

I have been working quite frequently on developing progressive web apps recently and I usually fire up a quick PWA with generators such as create-react-app or in this case using preact-cli. These generators have an advantage that all the hard service worker configurations are already done for me and I just need to focus on building the app logic while the starting point is already highly optimized for browser performance. We will see why this is not always a positive if we are not careful.

What are hidden files

These are files such as .DS_Store on macOS or Thumbs.db if you are on Windows. .DS_Store stores the directory properties such as the position of icons or background image of the folder, which is a generally a pretty useless file for most developers to even pay attention to and that makes it even more dangerous when it causes problems for us during a deployment.

Service worker caching

In a preact-cli inited PWA, the service worker will aggressively cache all assets and resources using the browser Cache API for faster loading. This happens as part of the automated build process. Preact is using the sw-precache library here to do most of the heavy lifting for us. The auto-generated files array looks like this:

var precacheConfig = [
        ["/app/index.html", "fa243faa5a7ba7dd91c2241a62010a67"],
        ["/app/not-exist/index.html", "14e3d8a57e9e195437405c1631413240"],
        ["/app/pnr/index.html", "741939de6641dfeb520d8f3bf8288192"],
        ["/app/schedule/index.html", "330df6ceb7147e93a2e3e6daec6acb72"],
        ["/app/static/about/index.html", "4accf5ab4d635725099660c1ec9b09fd"],
        ["/app/static/contact/index.html", "a4a1b570fdb015996fa4d9d52bf38733"],
        ["/app/status/index.html", "9a8ed5b95b2d6dbf4a637e084dedc99c"],
        // more files...

The .html are pre-rendered files generated by preact-cli which will be cached automatically for us to allow first load to be as fast as possible.

When caching goes wrong

So in this particular progressive web application, we have been deploying and registering the service workers just fine until I received a few emails from my client at 4am saying certain features like "Add to Home Screen" is broken and all the pages are being slow.

The first thing I do is to open up the web app and use the Chrome Lighthouse auditing tool to quickly find out potential problems with performance and service workers. This is what I see.

Ouch, a PWA score of 64, that is not good because I remember promising the client a score of at least 85. So something is clearly wrong with the last deployment. Let's scroll down a bit more to see:

Here is the issue. The service worker has not been installed correctly so none of its benefits are going to work in the application. The service worker must have encountered some kind of error during the installation process. The next step is to take a look at the Application tab to see what's going on with the service worker. Looks like we have an error log here as well.

Looks like there is an error that the DS_Store hidden file is not founded. Let's see the full sw.js file and prettify the javascript to see what's going on.

It seems that the build process wrongly assumed that this file should be cached and is therefore included in the list of files to be cached by the browser. So what is the problem? Well, during the deployment we rightfully ignore all hidden files such as this firebase.json indicates here:

  "hosting": {
    "public": "build",
    "ignore": [
      "firebase.json",
      "**/.*",
      "**/node_modules/**"
    ],

So the .DS_Store files will not be found and therefore will cause the service worker to not install correctly and rendering all our performance optimizations useless.

The solution to this problem was to remove the hidden files, rebuild, and redeploy the app. Now the service worker installs just fine. I might make a shell script that checks the existence of nasty hidden files before running the build as a further step to be further sure if I really wanted to make sure this does not happen again.

Conclusion

This caught me off guard as I have deployed this application several times already without any problems. One solution is to disable the automatic creation of DS_Store files on macOS.

I hope reading this will mitigate some potential frustrations for some of you or at least save some time finding the problem. Let me know if you came up with any good solutions or have come across similar issues in your work.