DEV Community: Steven Tey 👨🏻‍💻

Building A Word Association Network from Scratch

Steven Tey 👨🏻‍💻 — Tue, 05 Jan 2021 19:17:09 +0000

Ever since I first launched One Word Domains this summer, I've received plenty of requests for a search feature to help navigate the ever growing list of domains on the site (the current number is about 500K domains).

My first thought when I started planning the implementation of this feature was:

Oh, this shouldn't be too hard, I'm just gonna:

Get the list of words that are the most similar to the user's search query using the spaCy NLP library and display them on the site.
Also show the list of TLDs that are available for that particular query.

No biggie.

However, when I started translating my vision to code, I ran into a few technical constraints.

First, I wanted the search to be blazing fast, so calculating the cosine similarity in real-time is a no-go – that would simply take too long.

Also, even if I were able to optimize the calculation time to <50ms, how do I deploy the spaCy model to the web when even the en_core_web_md model itself is over 200MB unzipped?

It took me 3 whole days to figure out a viable solution for this, which is why I'm hoping this blog post will help you do so in much shorter time.

Building A Word Association Network

After doing a bunch of research, I came to the conclusion that there was only one way for me to reduce the search time to sub-50ms levels – by pre-training the model locally and caching the results in PostgreSQL.

Essentially, what I'd be doing is building out a word association network that maps the relationship between the 20K words that are currently in my database.

Picture this:

Note: the decimal values on each of the edges represent the similarity scores between the auxiliary terms and the root term

Now picture that again, but this time with 20,000 adjectives, nouns, verbs, and 10 other categories + French and Spanish words*.

I know, sounds pretty crazy and cool at the same time, right?

First, I installed the spaCy library in my virtual environment and downloaded the en_core_web_md model:

pip install spacy
python3 -m spacy download en_core_web_md

Then, I imported them into my Flask app and loaded up the word vectors:

import spacy
import en_core_web_md
nlp = en_core_web_md.load()

Now it's time to build the word association model. I first took the list of vocabs that were on One Word Domains and tokenized them. Then, by using a nested for-loop, I calculated the cosine similarities between each of the terms, preserved a list of the top 100 most similar auxiliary terms for each word using the addToList function, and stored everything in a dictionary. The code for all that is as follows:

# Add to list function to add the list of significant scores to list
def addToList(ele, lst, num_ele):
    if ele in lst:
        return lst
    if len(lst) >= num_ele: #if list is at capacity
        if ele[1] > float(lst[-1][1]): #if element's sig_score is larger than smallest sig_score in list
            lst.pop(-1)
            lst.append((ele[0], str(ele[1])))
            lst.sort(key = lambda x: float(x[1]), reverse=True)
    else:
        lst.append((ele[0], str(ele[1])))
        lst.sort(key = lambda x: float(x[1]), reverse=True)
    return lst

import json

# list of English vocabs
en_vocab = ['4k', 'a', 'aa', ...]

# tokenizing the words in the vocab list
tokens = nlp(' '.join(en_vocab))

# initiate empty dictionary to store the results 
en_dict = {}

# Nested for loop to calculate cosine similarity scores
for i in range(len(en_vocab)):
    word = en_vocab[i]
    print('Processing for '+ word + ' ('+ str(i) + ' out of '+ str(len(en_vocab)) + ' words)')
    for j in range(i+1, len(en_vocab)):
        prev_list_i = en_dict[str(tokens[i])]['similar_words']
        en_dict[str(tokens[i])]['similar_words'] = addToList((str(tokens[j]), tokens[i].similarity(tokens[j])), prev_list_i, 100)
        prev_list_j = en_dict[str(tokens[j])]['similar_words']
        en_dict[str(tokens[j])]['similar_words'] = addToList((str(tokens[i]), tokens[i].similarity(tokens[j])), prev_list_j, 100)

    with open('data.json', 'w') as f:
        json.dump(en_dict, f)

This code took forever to run. For 20,000 words, there were a total of 200,010,000 combinations (20000 + 19999 + 19998 +...+ 3 + 2 + 1 = 20001 * 10000 = 200,010,000). Given that each combination took about half a millisecond to complete, it wasn't a surprise when the whole thing took about 36 hours to execute completely.

But, once those gruelling 36 hours were up, I had a complete list of words along with 100 most similar words for each and every one of them.

To improve data retrieval speeds, I proceeded to store the data in PostgreSQL – a relational database that is incredibly scalable and powerful especially when it comes to large amounts of data. I did that with the following lines of code:

def store_db_postgres():

    with open('data.json', 'r') as f:
        data = json.load(f)

    for word in data.keys():
        db_cursor.execute("""INSERT INTO dbname (word, param) VALUES (%s, %s);""", (word, json.dumps(data[word]['similarity'])))
        db_conn.commit()

    return 'OK', 200

And, voilà! You can now traverse the millions edges in the word association network at minimum latency (last I checked, it was at sub-50ms levels). The search tool is live on One Word Domains now – feel free to play around with it.

*The French and Spanish words were trained on the fr_core_news_md and es_core_news_md models respectively.

Deploying to the Cloud

Here comes the tricky part. I knew that while I had over 20K of the most commonly-used English words in my word association network, there's always gonna be a chance where a user's query is not part of my knowledge base.

Therefore, I wanted to add an option for users to generate results on the fly. To do that, I needed to upload the spaCy model to the web so that I can build additional edges to the word association network in real-time.

However, this wasn't possible on Heroku given the 500MB slug size hard limit (my current slug size was already 300MB). I needed a more powerful alternative, and that's when I decided to go with AWS Lambda.

I had my first encounter with Lambda function s when I had to use them to host the NLP component of a movie chatbot that I helped built for my AI class this fall.

While Lambda functions were rather complicated to set up in the beginning, with the help of the handy Zappa library and tons of StackOverflow posts, I was able to build One Word Domains' very own lambda function that would find the most similar words for a given query and return them in JSON format.

I won't go too deep into the weeds about the deployment process, but this was the guide that helped me immensely. Also, here's the main driver function that would find the list of the top 100 most closely associated words for a given query:

@app.route('/generate')
@cross_origin()
def generate_results():
    # get vocab
    vocab_pickle = s3.get_object(Bucket='intellisearch-assets', Key='vocab')['Body'].read()
    vocab_only = pickle.loads(vocab_pickle)
    vocab_only = list(set(vocab_only))
    # add new query to vocab
    vocab_only.insert(0, query)
    # store vocab back to pickle
    serialized_vocab = pickle.dumps(vocab_only)
    s3.put_object(Bucket='intellisearch-assets', Key='vocab', Body = serialized_vocab)
    # do the rest
    tokens = nlp(' '.join(vocab_only))
    results = []
    for i in range(1, len(vocab_only)):
        if str(tokens[i]) != query:
            results.append([str(tokens[i]), tokens[0].similarity(tokens[i])])
    results.sort(key=lambda x: x[1], reverse=True)
    return results[:100]

Here, I'm storing the list of vocabs in the form of a pickle file on AWS S3, and whenever I have a new query, I use s3.get_obect to retreive the list and update them by inserting the new word into the list. Note that I'm using the @cross_origin() decorator from Flask's CORS library since I'll be calling on this Lambda function from my Heroku app.

And that's...pretty much it. All I had to do now was connect the AWS Lambda API endpoint to my original Heroku app to help generate a list of relevant words for a given search query that a user enters on the site. Here's the outline for that:

import requests
url = "https://my-api-endpoint.amazonaws.com/generate?query=" + query
response = requests.get(url)
similar_words = response.json()
print(similar_words)

Finally, the moment of truth – here's a Loom recording showing how real-time search works on One Word Domains.

And there you go – a fully-functional search algorithm built on top of a word association network containing 20,000 nodes and millions of edges.

Feel free to play around with the search tool at the top of every page on One Word Domains. If you have any feedback, or if you find a bug, feel free to send me a message via chat, contact page, email or Twitter – I'd love to help!

Building An iOS "App" With 30 Lines of Javascript

Steven Tey 👨🏻‍💻 — Mon, 30 Nov 2020 17:09:53 +0000

Apple's iOS 14 update 5 months ago introduced iOS widgets, which allow users to get timely information from their favorite apps at a glance.

This also opened up a whole new realm of possibilites for developers to use widgets as surrogates for native iOS apps, hence saving tons of engineering resources while still providing an app-like experience for their users.

And this is how I was able to create a fully-functional "app" in the form of an iOS widget for Recurrence using Javascript, a RESTful API endpoint, and the Scriptable app.

What is Recurrence

Recurrence is a lightweight affiliate management software that allows you to keep track of all your affiliate sales in one single platform.

I built Recurrence with the objective of helping startup founders launch their affiliate programs in a hassle-free, one-click fashion without breaking the bank.

Abiding by the principle of dogfooding, I've been using Recurrence to track affiliate sales for my other project, One Word Domains, and the results have been super encouraging so far – over 456 clicks tracked and $275.65 in affiliate sales recorded.

However, the only way my affiliates were able to track the progress of their clicks was through the Recurrence web-app (pictured below), which was more of a desktop-first experience.

While the web-app was optimized for mobile, I still wanted something that was more mobile-friendly. Something that was native to the iOS experience.

And that's when I came across this tweet by Pieter Levels.

Turns out, all you needed to do to build an iOS widget were 3 simple steps:

Set up an API endpoint that contains the live data
Download the Scriptable app
Design and code the widget using Javascript

Building the API Endpoint

Since I was using the Flask framework, all I needed to do is add an extra route for the API endpoint:

from flask import Flask, jsonify
from flask_cors import cross_origin
import psycopg2

@app.route('/api/v1/<api_key>')
@cross_origin()
def fetch_data_api(api_key):
    db_cursor.execute("SELECT clicks, earnings FROM affiliates WHERE api_key = %s", (api_key,))
    data = db_cursor.fetchall()[0]
    clicks = data[0]
    earnings = data[1]
    data = {
        "stats": [{
            "clicks": clicks,
            "earnings": "{:.2f}".format(earnings)
        }]
    }
    return jsonify(data)

There are a few things to note in this code block:

I'm using @cross_origin(), which is a decorator from the flask_cors library that allows cross-origin browser/API requests for flask apps. You might need to install the library with pip install flask_cors for this to work.
Since I'm using PostgreSQL as my database, db_cursor is my psycopg2 object, which allows me to connect to the database from a remote origin. You might change this depending on your database schema.
This function returns the data in JSON format using flask'sjsonify method. This step is key, since the Scriptable app only reads data in JSON format.

Downloading Scriptable

You can download the Scriptable app on the App Store here. However, for local development, I'd highly recommend using the Scriptable Mac Client – that way you can test your code directly on your laptop instead of copying it from VSCode/Sublime and pasting into your phone just to test it out.

Once you've opened the Scriptable Mac App, all you gotta do is click on the little "+" sign at the top right corner and start coding up your Javascript.

Designing and Coding the Widget

First, let's build the function that will fetch us some live data from the API endpoint that we built earlier:

async function loadData() {
  let url = "https://recurrence.app/api/v1/YOUR_API_KEY"
  let req = new Request(url)
  let json = await req.loadJSON()
  return json.stats
}

Here, we're building an asynchronous function with the Request() constructor from the Fetch API. This function visits our endpoint and fetches the data in JSON format every minute or so (more on this later).

Then, we'll design the function that renders the frontend of our widget:

async function createWidget(data) {

    let widget = new ListWidget()
    widget.backgroundColor = new Color("#fff")
    widget.addSpacer()

    let stats = String(data[0].clicks) // In this case, we're getting the first element of the data object and the "clicks" child element.
    let stats_num = widget.addText(stats)
    stats_num.font = Font.boldSystemFont(50);
    stats_num.textColor = new Color("#4f5ce2")
    stats_num.centerAlignText()

    let subtitle = widget.addText("total clicks")
    subtitle.font = Font.mediumSystemFont(15);
    subtitle.textColor = new Color("#000")
    subtitle.centerAlignText()

    let image = widget.addImage(img)
    image.centerAlignImage()
    image.imageSize = new Size(100,35)

    return widget
}

This function creates a widget instance with new ListWidget() and begins defining various design parameters to style the widget:

Set the background color of the widget to white with widget.backgroundColor
Add the data that was retrieved from the API endpoint with widget.addText(stats)
Style the displayed data with stats_num.font, stats_num.textColor, and stats_num.centerAlignText()
Define and style the subtitle (# of clicks, total earnings, etc.) with the subtitle object
Add the logo of the app at the bottom with widget.addImage(img).

You can also refer to Scriptable's official documentation for more information about widget styling.

Finally, we'll have to call the functions we created earlier and render it in the Scriptable app:

const i = await new Request("https://i.ibb.co/cNqLLbD/logo.png");
const img = await i.loadImage();
let data = await loadData()
let widget = await createWidget(data)
if (config.runsInWidget) {
  // The script runs inside a widget, so we pass our instance of ListWidget to be shown inside the widget on the Home Screen.
  Script.setWidget(widget)
} else {
  // The script runs inside the app, so we preview the widget.
  widget.presentMedium()
}
Script.complete()

And that's it – you're all set! Go ahead and hit the "Run" button at the bottom right corner of the screen:

...and, voilà! Now all you gotta do is transfer this code over to the Scriptable app on your phone the same way you did on the Mac client. Here's the complete code snippet:

async function loadData() {
  let url = "https://recurrence.app/api/v1/YOUR_API_KEY"
  let req = new Request(url)
  let json = await req.loadJSON()
  return json.stats
}

async function createWidget(data) {

    let widget = new ListWidget()
    widget.backgroundColor = new Color("#fff")
    widget.addSpacer()

    let stats = String(data[0].clicks)
    let stats_num = widget.addText(stats)
    stats_num.font = Font.boldSystemFont(50);
    stats_num.textColor = new Color("#4f5ce2")
    stats_num.centerAlignText()

    let subtitle = widget.addText("total clicks")
    subtitle.font = Font.mediumSystemFont(15);
    subtitle.textColor = new Color("#000")
    subtitle.centerAlignText()

    let image = widget.addImage(img)
    image.centerAlignImage()
    image.imageSize = new Size(100,35)

    return widget
}

const i = await new Request("https://i.ibb.co/cNqLLbD/logo.png");
const img = await i.loadImage();
let data = await loadData()
let widget = await createWidget(data)
if (config.runsInWidget) {
  // The script runs inside a widget, so we pass our instance of ListWidget to be shown inside the widget on the Home Screen.
  Script.setWidget(widget)
} else {
  // The script runs inside the app, so we preview the widget.
  widget.presentMedium()
}
Script.complete()

Adding Widget to Home Screen

Lastly, you'll have to add your widget to your iPhone home screen. The fastest way to do that in iOS 14 would be to:

Tap and hold on your home screen until your apps begin to jiggle
Click on the "+" button at the top right corner of the screen
Select the Scriptable widget and your desired widget size
Click on "Add Widget"
Long hold on the widget again and select "Edit Widget"
For the "Script" selection, choose the script you pasted in earlier
For the "When Interacting" selection, choose "Open URL" and key in the URL for your app– in this case that would be https://recurrence.app/dashboard. This will allow you to get redirected to your app when you click on the widget.

Here's a quick GIF showing the steps above:

In most cases, the widget usually updates every minute or so, but if you want to receive notifications about the changes, you can set that in the Scriptable app under "Notifications".

If you did everything right, you should end up with the following widgets on your home screen:

Congratulations, you've just built your first iOS widget! 🥳

Feel free to comment below or ping me via email or on Twitter if you run into any issues along the way – good luck!

Building A Lightweight Blogging CMS In 10 Lines of Code

Steven Tey 👨🏻‍💻 — Fri, 06 Nov 2020 19:33:55 +0000

Ever since launching One Word Domains five months ago, I've been wanting to set up a blog where I could:

Document my build process
Write about some of the coolest programming tips and tricks that I learned in the process (this blog post is one of them)
Share some insights on the domain industry - i.e. what are some of the top naming conventions in Silicon Valley

However, I quickly ran into a dilemma when trying to find a suitable blogging CMS (content management system) for my needs:

Wordpress is easy to set up, but is also an overkill - I don't need 15 different subpages and a full-fledged user management system (I'm already using PostgreSQL for that)
Ghost is a little more challenging to set up (here's a great guide if you're into that) but would require setting up an extra dyno on Heroku or a VPS on Digital Ocean - which would mean an extra $5 - $7 a month
Medium is relatively pain-free to set up, but is pretty limited when it comes to customization + you're not really helping your site's SEO with your posts since you'll be contributing to Medium's SEO instead

What I was looking for was a simple and free static-site solution that was easy to customize + integrates well with my existing stack (Heroku, Flask, PostgreSQL, Python, HTML/CSS, JavaScript, jQuery).

I decided to consult my friend Linus, who recommended the Python-Markdown library - which is the same framework that Pelican (the Python version of Hugo) uses.

Intrigued, I began researching the origins of the Python-Markdown library, and that's when I came across this blog post by James Harding. 10 lines of code later, I've successfully set up my very own Markdown-powered static site for the One Word Domains Blog.

Here's how everything went down, step by step:

Requirements

First, I imported the Flask-FlatPages and Markdown libraries:

import markdown
from flask_flatpages import FlatPages

...and declared them in my requirements.txt file:

Flask-FlatPages==0.7.1
Markdown==3.2.1

Folder Structure

Since I already had an existing Flask app up and running, all I did next was add a /posts folder at the root directory, a separate folder called blog-images under the /static/assets folder, and a few template files in the /templates folder. Here's a rough overview on how my folders were structured:

  ├──app.py
  ├──posts
  │       └──post1.md
  │       └──post2.md
  ├──templates
  │       └──blog.html
  │       └──post.html
  └──static
        └──assets
        │     └──blog-images
        └──script
        └──styles

Define FlatPages ENV Variables

Before I started setting up the Flask routes for my blog, I defined the ENV variables for FlatPages in my app.py file, right after initiating the Flask app:

FLATPAGES_EXTENSION = '.md'
FLATPAGES_ROOT = ''
POST_DIR = 'posts'

flatpages = FlatPages(app)
app.config.from_object(__name__)

Here, I defined FLATPAGES_ROOT as '' because the folder containing all my markdown files, posts, is located in the root directory – which is why POST_DIR is defined as 'post'.

Flask Routes

Here are the 10 lines of code that I mentioned earlier – which I inserted into my app.py file:

@app.route("/blog")
def blog():
    posts = [p for p in flatpages if p.path.startswith('posts')]
    posts.sort(key=lambda item:dt.strptime(item['date'], "%B %d, %Y"), reverse=True)
    return render_template("blog.html", posts=posts)

@app.route("/blog/<permalink>")
def blog_post(permalink):
    path = '{}/{}'.format('posts', permalink)
    post = flatpages.get_or_404(path)
    return render_template('post.html', post=post)

I know, I couldn't believe it either.

10 lines of Python code was all I needed to get the One Word Domains Blog up and running.

Let's dive deeper into the lines of code above and see what each one of them does:

The first route, /blog hosts the landing page of the blog. Here, the code iterates across all the Markdown files present in the /posts folder and interprets them in the form of a flatpages object. It then sorts them in descending order by published date – here, I'm using the dt.strptime() method because my dates are written in natural language format (October 30, 2020). Lastly, the code renders the blog.html template and sends over all the posts as jinja variables.
The second route, /blog/<permalink> takes care of the individual blog posts. The first line of code creates the composite path for each of the Markdown files, which is in the format /posts/post1.md. It then gets the files with the flatpages module and renders the post.html template along with all the attributes of the particular blog post.

Markdown Format

Let's take a look at the format of a given Markdown file, say, the one for this blog post, for example:

title: Building A Lightweight Blogging CMS In 10 Lines of Code
subtitle: This is the full story of how The One Word Domains blog was built - with 10 lines of Python code, the Flask-Flatpages library, and a bunch of Markdown files.
date: November 2, 2020
image: post2-thumbnail.png
permalink: markdown-flask-lightweight-cms

Ever since launching One Word Domains five months ago... (content)

As you can see, each Markdown file has the following attributes:

title: The title of the blog post
subtitle: The subtitle, or 'tagline' of a blog post, usually written to give more context on the post
date: The date the blog post was published
image: The thumbnail image for the blog post, stored within the /static/assets/blog-images folder that I mentioned earlier
permalink: The canonical URL for the blog post. Protip: try and use dashes and keep this below 74 characters so that it doesn't get truncated in the search results
content, or html: The bulk of the blog post's content

HTML Templates

Here's a rough outline of my blog.html template:

{% for post in posts %}
<a href="/blog/{{ post.permalink }}">
   <img src="/static/assets/blog-images/{{ post.image }}"/>
   <h1>{{ post.title }}</h1> 
   <p>{{ post.date }}</p>
   <p>{{ post.subtitle }}</p>
</a>
{% endfor %}

This code will iterate across all the Markdown files in the /posts folder that I set up earlier and auto-generate previews for each and every one of them.

And here's the one for my post.html file:

<img src="/static/assets/blog-images/{{ post.image }}"/>
<h1>{{ post.title }}</h1> 
<p>{{ post.date }}</p>
{{ post.html|safe }}

Compile and Run

If everything went well, your blog should be live at 127.0.0.1:5000/blog once you run $ python app.py in your terminal. Yay!

Or, if you're like me and you run into a bunch of errors in your first few attempts - don't give up! Debug your code by pasting the error messages into Google and clicking on the first Stackoverflow post that pops up.

Good luck!

Bonus: Typora

I first started editing my Markdown files in Sublime, which was rather mechanical and cumbersome.

Then, everything changed when the fire nation attacked I discovered this free tool, Typora (or at least, "free during beta", as stated on their site). The intuitive and seamless writing experience that Typora provides is unparalleled, and while **this is not an ad, I highly recommend trying it out.