DEV Community: Algolia

Setting up your MacBook for mobile Flutter development

Chuck Meyer — Tue, 20 Dec 2022 19:53:14 +0000

This month I'm trying to level up on my mobile development skills by learning Flutter with and end goal of building a mobile search application. The first step was setting up a functional local dev environment on my laptop. I realized that there would be a few steps, but I didn't anticipate needing SIX different languages to get up and running!

In this post, I'll walk you through setting up your MacBook for cross-platform Flutter development on iPhone and Android devices.

Assumptions

You will need a terminal configured just the way a like it and the brew package manager installed. If you don't have it already, you should install brew first.

Install Flutter

The easiest way to install flutter is using brew

brew install --cask flutter

Once brew has installed flutter and all of its dependencies, you can use flutter itself to track what's left to do:

flutter doctor

This is also a good time to install command completion for flutter. Here's the command to that for zsh:

flutter bash-completion >> ~/.zshrc

Depending on what you already had installed, you probably have a lot left to do, so let's get to it!

The iOS build path

For iOS development, you need a full version of Xcode including the iOS SDK, not just the command line tools. It's easiest to install this from the App Store. Once it's finished installing, drop back to the command line and run the following command to accept the License agreement and install any additional components.

sudo xcodebuild -runFirstLaunch

You'll also want cocoapods for dependency management on the iOS side. Install it using brew.

brew install cocoapods
pod setup

That should be everything you need for iOS. Let's test to see if it works.

Testing the iOS build path

It's time to launch an iPhone simulator and create your first Flutter project.

open -a Simulator
flutter create my_first_app
cd my_first_app
flutter run

This should launch an iPhone emulator, create the default click-counting Flutter app, and load it onto the emulated iPhone.

The Android build path

On the Android side, you'll want to install Android Studio and the Android SDK. This guide walks you through the steps, but here's an overview.

First, download and install Android Studio for MacOS.
Launch Android Studio and create your first project.
Select Tools -> SDK Manager -> Android SDK -> SDK Tools and install the Android SDK Command-Line Tools and the latest Android Emulator

Switch back to the command line to accept all necessary licenses for Android using the following command.

flutter doctor --android-licenses

You'll need a Java runtime environment if you didn't already have one installed. The easiest is to just install the latest JRE form Oracle.
That should do it for the Android side. Time to test.

Testing the Android build path

To test the Android side, install an emulator and set it up as a target for flutter. First, choose an image.

~/Library/Android/sdk/tools/bin/sdkmanager --list | grep system-images

Select a recent system image to install and create an Android Virtual Device. Then you can launch the emulator and load the click-counting Flutter app.

~/Library/Android/sdk/tools/bin/sdkmanager --install "system-images;android-33;google_apis;arm64-v8a"
~/Library/Android/sdk/tools/bin/avdmanager create avd -n "pixel_5_api33" -k "system-images;android-33;google_apis;arm64-v8a" -b "arm64-v8a" -d "pixel"

flutter emulators –-launch pixel_5_api33
flutter run

You are now ready to begin cross-platform development of your mobile app!

Next time, we'll add search to our mobile applications using the Algolia Flutter Helper.

DevTool Intro: The Algolia CLI!

Khalid Elassaad — Mon, 15 Aug 2022 19:11:00 +0000

Wake up Neo...

We need you to reset the configuration for our pharmaceutical products index! We were messing around with the highlighting settings, and now results for the query “red pills” are highlighted blue, which is really very confusing…

Also, we’re a little concerned about the possibility of all experienced reality being a simulation run by machines to keep humanity enslaved!

Please fix the index first though!

—

Oh man, don’t you all just love getting woken up before sunrise on a Monday for stuff like this? In an ideal world, this would be a quick fix. With a flourish of my black trench coat, I would roll out of bed and land in a crouch in the middle of the room, laptop in hand. I’d press enter to skip the white rabbit and ignore the knocking at my door. With one hand, I’d adjust my too-dark pince-nez glasses while the other hand fires off a command in seconds. Something short and sweet and to the point. Something like this…

algolia settings import prod_pharmaceuticals_index -F pharm_settings_snapshot.ndjson

A glance at the terminal reveals the cheerful response:

✓ Imported settings on prod_pharmaceuticals_index

Nice… Now what was that other thing about machines enslaving humanity?

—

Meet the Algolia CLI!

If you’re anything like me, the command line is your friend and trusted ally. Whether I’m testing a quick API call or coding in my favorite IDE, working with the command line keeps me iterating at a smooth clip. Additionally, I love that anything I do through a command line interface (CLI) can be easily scripted and scheduled to run however and whenever I like. With a strong set of tools in my CLI toolbelt, solving problems and automating solutions is a walk in the park.

That’s why we are so excited to announce the Public Beta launch for the brand new Algolia CLI Tool, with the full release coming soon!

Algolia CLI makes uploading an index, automating common dashboard operations, or saving and reloading snapshots of your configurations, possible right from the command line! No API client needed!

So how do I start? Don’t worry, you won’t have to visit the oracle for this one… Getting started with the Algolia CLI is a breeze!

If you’re on MacOS, simply use homebrew to install the tool by running this command in terminal:

brew install algolia/algolia-cli/algolia

Want to poke at the code and build it yourself? The Algolia CLI is fully open-source, under an MIT license, and lives in a public github repository (which you can find here)!

Releases for Linux and Windows coming soon!

Let’s take a look at some specific use cases to see how the Algolia CLI can make your life easier, so you can focus on saving the world, you star!

—

Example #0: Set your active Algolia application with profiles

Any Algolia CLI command can be called with the --admin-api-key [string] and --application-id [string] flags to specify which application to interact with. To enable you to fire off commands without having to lug around api-keys and app-ids in your clipboard all the time, you may use the following command to register a default profile.

algolia profile add --name [string] --app-id [string] --admin-api-key [string] --default

✓ Profile 'test_pharm_app' (##APP#ID##) successfully added and set as default.

The --default flag tells the CLI which app to use with future commands when no profile or API key/app ID pair are provided. Only one application can be set as default at a time. Taking time to register profiles makes future interactions fluid, especially when working with 2 or more Algolia apps.

You can also interactively provide the fields for this command by running:

algolia profile add

The tool will then prompt you for each field, one at a time, and add the application as specified.

To see a list of all added applications, simply run:

algolia profile list

NAME                  APP ID      NUMBER OF INDICES  DEFAULT
test_pharm_app        #APP#ID#1#  2                  ✓
playground_pharm_app  #APP#ID#2#  1
prod_pharm_app        #APP#ID#3#  15

Awesome! We see here that we have multiple Algolia apps added, and that the app test_pharm_app is our default!

Note: app profiles are saved in ~/.config/algolia/config.toml

Let’s upload some data!

—

Example #1: Create an index and upload records from a file

I want to create a new index, give it a relevant name, and upload the records from a file on my local system. Can the CLI handle that? Of course it can!

algolia objects import new_index_name -F ./path_to_file.ndjson

Did you notice that we didn’t specify an app ID, api key, nor app profile? In this case, the CLI will act upon the default application (which is test_pharm_app as indicated in the previous step).

We can also pass content to the CLI for upload via stdin by replacing the -F flag with -. Let’s pipe some commands together! The following command produces the same result as the previous one(read file and upload contents to specified index), but uses cat to pass the contents of the file to the CLI tool’s stdin as opposed to passing the file path as an argument. Neat!

cat ./path_to_file.ndjson | algolia objects import new_index_name -F -

What is ndjson? Newline delimited JSON is the format the Algolia CLI reads from and writes to files. This means that any command that passes ndjson formatted data as output or accepts it as input can be piped together with an Algolia CLI command! We’ll see more of this in the next example

–

Example #2: Managing index settings snapshots

So far, we’ve connected our CLI to our Algolia apps, and we’ve uploaded some data to an index! Let’s take it a step further by snapshotting an index’s settings so we can restore them to a healthy checkpoint in the future.

Which indices exist in our default app, test_pharm_app?

algolia index list

NAME               ENTRIES  SIZE    UPDATED AT  CREATED AT  LAST BUILD DURATION  PRIMARY  REPLICAS
pills_treatments   1,000    100 kB  1 day ago   1 day ago   2s                            []
pills_cures        0        0 B     2 days ago  2 days ago  3s                            []

Let’s save the settings for index pills_treatments to a file on our system.

algolia settings get pills_treatments > ./pt_settings_snapshot.ndjson

Snapshot created! Now, reverting to the snapshot is as easy as…

algolia settings import pills_treatments -F ./pt_settings_snapshot.ndjson

✓ Imported settings on pills_treatments

We can also transfer settings from one index to another in one line! Let’s copy the settings for the pills_treatments index over to the pills_cures index! In the import command, we’ll use the - value after the -F flag to tell the CLI to read input from stdin instead of a specified file.

algolia settings get pills_treatments | algolia settings import pills_cures -F -

✓ Imported settings on pills_cures

Easy! One more scenario… we’ve tested this out on all our indices in the test app, now we’re ready to migrate these settings to our production app! We’ll target the pills_treatments_prod index in the prod_pharm_app profile.

algolia settings get pills_treatments | algolia settings import pills_treatments_prod -F - -p prod_pharm_app

✓ Imported settings on pills_treatments_prod

You’re beginning to believe, aren’t you?

—

Example #3: CI/CD pipelines that execute Algolia tasks

This is where things start to get really exciting. Consider the following workflow:

A database contains your data.
An automated script runs nightly to generate an index from the data in the database.
An automated script uploads that index to Algolia. Neat! Cool! Convenient! Let’s change that. We’re gonna mess things up a bit. How about we push a bug to prod?

Engineer Anderson is tasked with performing a database migration. He creates a change!
The change is reviewed by Engineer Becky, who says LGTM!
The change is submitted!
The CI/CD pipeline applies the change to the test database. No errors! Database migration tests pass!
The change is applied to the prod database. No errors! Database migration tests pass!
Engineer Anderson signs off with a smile on his face, goes home, and helps his landlady take out the garbage. …

That night, while Engineer Anderson is sleeping peacefully and dreaming about dangling below a helicopter in flight, disaster strikes!

…

The automated index script runs at night, generating an index from the data in the database.
However, the database migration transformed fields consumed by this script.
The output index is missing an important search attribute, but uploads successfully.
Searching the malformed index produces inaccurate and irrelevant results. Oh no!
It isn’t until 3 days later, as search metrics drop, that the problem is even detected. It takes Engineer Anderson another day to roll back the changes and fix the index. What can be done about this? How do you prevent this mess from ever happening again?

What if DB changes ran a few simple query test cases during the CI/CD pipeline? Each change would immediately generate an index and fire a series of test queries against it. Engineer Anderson’s change would have failed the CI/CD pipeline for producing different query results than expected. All it would take is the creation of a handful of test cases and a simple script to run the Algolia CLI’s algolia search command against those test cases to detect this issue in advance!

Let’s configure one such test case! When the index is working properly, we can execute a test query and save the results. The saved data is our “expected results”, against which we’ll compare the results of subsequent query tests. We’ll search an index from the previous example for “blue pills”:

algolia search pills_treatments_prod -p prod_pharm_app --query "blue pills" > blue_pills_expected_results.ndjson

Cool! We have a test case for the query “blue pills”! Let’s run this on the pills_treatments index in the test app, test_pharm_app, and see if it matches.

algolia search pills_treatments -p test_pharm_app --query "blue pills" > blue_pills_actual_results.ndjson && diff blue_pills_expected_results.ndjson blue_pills_actual_results.ndjson | wc -m

We query the test index and save the results. Then, we compare the expected results with the actual results by invoking the diff command, which outputs only the difference between the two files. Finally, we pipe the output of diff to the wc -m command, which counts the number of characters in the output from diff. We observe an output of 0, meaning there is no difference between the two files, which indicates that our query produced the same results in both indices and that our query test passed!

Along those lines, a non-zero output would mean that there is some difference in the two files, and our query test failed to produce identical output across both indices.

This is just one example of how you could integrate the Algolia CLI within a CI/CD pipeline. Imagine using the index settings management workflow to promote dashboard-configured settings set by your marketing team to a production app. The promotion can then invoke a run of query testing to validate the changes, and voila! The CLI becomes a force multiplier for your business users as well!

Use the Algolia CLI to automate tasks, improve your workflows, and keep your search experiences running seamlessly, even as you change and develop the content under the hood!

—

Get Involved!

We’ve seen that the Algolia CLI can handle almost anything, from simple operations like index upload to more complex workflows like snapshot restoration. Stay tuned as we push updates out and further expand the CLI’s capabilities to make your life as an Algolia developer even easier!

Remember, the Algolia CLI is in Public Beta! On MacOS, install with Homebrew:

brew install algolia/algolia-cli/algolia
Windows and Linux installers coming soon!

Or, clone the GitHub repo and build it yourself!

Want to learn more? Check out the Official CLI Documentation for a guided tutorial!

And don’t forget to mark your calendars for September 14-15! The Algolia DevCon fast approaches. We’ll be spotlighting the CLI tool and demonstrating its capabilities! The Developer Experience team will be there to host live Q&A and answer all your questions. You won’t want to miss it!

—

Deprecation Advisory - Before this CLI was created, we had an npm Algolia CLI package that served similar functionality. The Legacy CLI is DEPRECATED - We will not be maintaining/updating/supporting it going forward. However, we will not remove the package from npm (since customers are still using it). Instead, we’ve announced the deprecation in the Legacy CLI GitHub repo (and renamed the repo to “algolia-cli-old”), and are redirecting customers to the New CLI GitHub repo.

Customize Algolia React components with Tailwind classes

Chuck Meyer — Tue, 19 Jul 2022 19:46:45 +0000

While I was at Algolia's Paris office last week, Dhaya Benmessaoud from our Front-end Experience team showed me a nifty trick for styling Algolia's React widgets in your UI. Out-of-the-box, Algolia provides a couple of pre-built themes for search experiences (Algolia and Satellite) and the ability to create custom themes. Recently, the front-end experience team has added a third way to style your UI by injecting custom CSS classes into your Algolia React components.

This is excellent news for people who use class-based CSS frameworks like Bootstrap and Tailwind! In my case, I was working with the Algolia Ecommerce UI Template, which relies on Tailwind for styling. I wanted to add a <TrendingFacets> widget from the Algolia Recommend UI library to my homepage, but I wanted to style it using Tailwind classes to match the rest of my front-end.

Here's how it looks before styling:

I can definitely do better! To accomplish this, I need to use the classNames attribute for my component. It's available for all of Algolia's React widgets (including Recommend) and allows you to override styling on component-specific elements. Some of our other front-end APIs like Vanilla JavaScript (cssClasses) and Vue (class-names) have had this functionality for years, and now it's available in React thanks to the recent refresh that added React hooks.

The documentation lists the elements I can override for each Algolia widget. For instance, here are the elements for a <SearchBox> widget:

root: The root element of the widget.
form: The form element.
input: The input element.
submit: The submit button.
reset: The reset button.
loadingIndicator: The loading indicator element.
submitIcon: The submit icon.
resetIcon: The reset icon.
loadingIcon: The loading icon.

For my <TrendingFacets> widget, I want the list in a horizontal line to conserve space, so I add a flex class to its list element. I'll also add a new facetItem class to give each item a clean capsule shape with some simple hover styling. Here's how my component looks after styling.

And here's the code:

<TrendingFacets
  classNames={{ list: 'flex', item: 'facetItem' }}
  recommendClient={recommendClient}
  indexName={indexName}
  maxRecommendations={3}
  itemComponent={({ item }) => (
    <a href={item.facetValue}>{item.facetValue}</a>
  )}
  facetName={facetName}
/>

Adding classNames to customize the style of Algolia widgets makes so much sense, especially for people like me who are addicted to Tailwind CSS for styling front-ends. You can read more about adding custom CSS classes to widgets in the Algolia documentation. If you're new to Algolia, you can try it out by signing up for a free tier account.

I'd love to see other examples of well-styled search experiences in the comments!

Feature Spotlight: Trends Models in Recommend

Chuck Meyer — Thu, 07 Jul 2022 03:37:03 +0000

Recommendations thrive on data – the more you know, the better the recommendations you can make. Algolia’s Recommend product puts your data to work using machine-learning to suggest products or content that will interest your customers. Today, we’re putting a spotlight on Recommend’s new Trends models. You’ll learn about how they work and how they expand the ways that you can use recommendations.

At launch, Recommend offered two initial machine-learning models: Related Products and Frequently Bought Together. Both are collaborative-filtering models, meaning you train them on the products your users click on and purchase as they interact with your website (events data). These models are great for product-centric recommendations. Given a specific product from your catalog, you can use the models to suggest other products that your customers either found similar or also purchased. You can place these recommendations on product detail pages and checkout flows to help your customers discover other products they might be interested in. Separate to what we'll be discussing in this article, these initial models have just been updated and you can now get started using them right away without events data - learn more here.

Trends models are similarly powered by collaborative-filtering, but are different in that they aren’t restricted to recommendations based on a single product. Instead, they look for product trends across the entire catalog, or within a particular facet/category of your catalog.

Diving into Trends

To use the Recommend models, you’ll need to have your products stored in an Algolia index. Then, you can track trends for either items or facet values across your index. Remember that a facet is an attribute, like “color”, that you have configured in your index for filtering. The possible values for this attribute across your index are called facet values – like “red” or “purple”.

Depending on the type of trend, you will train one of two new models:

Trending items - The most popular items either for a specific facet value or across the entire catalog.
Trending facet values - The most popular values for a specific facet (e.g. “sweaters” as a value for “category”).

Both models are trained by collaborative-filtering, meaning they learn trends from events data – specifically conversion events captured from your users. A conversion event could be purchasing an item, adding it to a shopping cart, or marking it as a favorite.

The trending items model looks at both the total number of conversion events for each item over the last few days, and the change in the number of conversion events over time to learn “trendiness” for each record in the index. It then assigns a score to each item in the catalog, either globally or within a particular facet value.

For trending facet values, this model does the same thing for a facet attribute – scoring the total number of conversions and change in conversions over a number of days for each facet value. You can collect trend information on up to three facets per model.

In order to use trends, you’ll need to train the model with at least 500 conversion events spread out over at least three days. Obviously, the more information you send, the better the recommendations the model can provide.

Where to use Trends

Trending items complement the other product-centric models by allowing you to recommend popular items or top sellers within a particular category on your home page before your customers have selected a single product.

Trending facets are great for adding popular categories or product attributes to your homepage that are relevant right now – whether it’s garden supplies in the spring or coats and hats in the fall.

Both models, once trained, can provide up to 30 recommendations, either globally or within a specified facet.

Training your Trends model

Before you can use any of the Recommend models, you’ll need to sign-up for an Algolia account and load your product catalog into an index. You can use an existing Search index or set one up just for Recommend.

Once your index is populated with product records, you’ll need to start sending user event information to Algolia. You can use the event integrations built into Algolia’s InstantSearch front-end libraries, the Algolia Insights API, or supported integrations with major ecommerce platforms and analytics back ends to collect this data. You can even upload your events as a CSV file with the appropriate record format.

Remember, because the Trends models look at changes over time, you’ll need to send conversion events that span multiple days.

The minimum event requirements are:

500 conversions in the last 30 days
Events across a minimum of 3 days (so not 500 conversions in 1 day)
Events for a minimum of 10 items in your index

You can track your events totals across your application in the Data Sources section of the Dashboard. This section also gives you access to debugging tools for troubleshooting your events.

To start training your model, choose the Trends model from the Recommend Dashboard. Select your product index and Recommend will let you know if you have enough of the required conversion events to start training. If you see an error about retrieving events, you may need to disable your ad-blocking software.

Next, you can select any facets that you want to take into account when training your model. For instance, you could train the model for Trending facet values on the facet category_page_id. You are now ready to click Train Model to start the training process. Training typically takes a few hours to complete.

Add trends to your frontend using the Recommend UI

Once your model is trained, you are ready to start adding Trend recommendations to your application. The easiest way to do this is with the Recommend front end UI libraries for vanilla JavaScript or React.

Let's say that you want to add trending categories to the Algolia Ecommerce UI Template using your newly trained Trending facet values model. First, you’ll need to add the recommend and recommend-react libraries to your project using npm (or yarn):

npm install --save @algolia/recommend-react @algolia/recommend

Once the libraries are installed, you’ll need to write a new FacetList component to retrieve and display facet recommendations from your trained model. Your component needs to initialize a Recommend client using the App ID and API key from the app that contains your trained model.

const recommendClient = recommend(ALGOLIA\_APP\_ID, ALGOLIA\_SEARCH\_ONLY\_API\_KEY)

Next, add the TrendingFacets widget from the Recommend React library. Since you’re working in the Ecommerce UI, you’ll use the abstracted Container component as a wrapper and handle titles the same way as other components in this library. That means you’ll want to suppress the default header by including headerComponent={() => null}. You can avoid overwhelming your visitors by reducing the list of 30 recommendations to just the top 3 using maxRecommendations={3}.

Here’s the complete FacetList component:

import algoliasearch from 'algoliasearch';
import recommend from '@algolia/recommend'
import { TrendingFacets } from '@algolia/recommend-react'
import classNames from 'classnames'
import { Container } from '@/components/container/container'

const recommendClient = recommend(ALGOLIA\_APP\_ID, ALGOLIA\_SEARCH\_ONLY\_API\_KEY)

export type FacetListProps = {
  title?: string
  indexName: string
  facetName: string
  className?: string
  \[index: string\]: any
}

export function FacetList({
  indexName,
  facetName,
  title,
  className,
}: FacetListProps) {
  return (
    <section className={classNames('py-4 laptop:py-16', className)}>
      <Container>
        {title && (
          <h2 className="text-sm font-semibold tracking-\[2px\] uppercase mb-3 laptop:mb-6 laptop:ml-3 laptop:heading-3">
            {title}
          </h2>
        )}
        <TrendingFacets
          recommendClient={recommendClient}
          indexName={indexName}
          maxRecommendations={3}
          itemComponent={({ item }) => <a href={item.facetValue}>{item.facetValue}</a>}
          headerComponent={() => null}
          facetName={facetName}
        />
      </Container>
    </section>
  )
}

Finally, add the finished component after the Banner widget on your index.tsx page:

      <FacetList
        title="Trending Categories"
        indexName="prod\_ECOM"
        facetName="category\_page\_id"
      />

After reloading, you should now see the top trending categories, just below the banner on the home page:

Conclusion

Trends models are a great complement to the existing, (and newly updated), Product models in Recommend. You can combine both types of models to help your customers discover new items during their entire journey on your site – facet value trends on the homepage lead them to trending items on the category page, related products on a product detail page, and finally to items frequently bought together during checkout. At each stage, customers are guided to items that they may have missed through a traditional search.

You can start using Trending items and Trending facet values right now from the Algolia Recommend dashboard. Or, you can read more about the other features that Recommend has to offer.

Feature Spotlight: Optional Filters

Chuck Meyer — Mon, 18 Apr 2022 16:40:29 +0000

While answering Algolia forum posts last week, I did a deep dive on Optional Filters for Algolia Search. Similar to filters and facet filters, optional filters are applied at query-time, allowing you to use all sorts of contextual information to improve results. Unlike the other filters, optional filters don't remove records from the result set. Instead, they allow you to say, "If records match this filter, move them up or down the ranking," without changing the number of records returned.

Some interesting use cases:

Move posts that mention a particular user to the top of the result set
Up-rank products available at a customer's local store
Down-rank products that are out-of-stock
Pin a specific record to the top of the listing based on an external recommendations engine

You can use filters and facet filters to reduce the number of records in the result set at query-time, then optional filters to manipulate the rankings for the remaining records. You can even apply filter scoring to control the order of the records further.

Here's an example that applies facet filters for product_type and price_range to an index from a Shopify store. The code selects the objectID of two records to promote, then injects them as optional filters into the query. If either of those products is part of the result set that matches the other criteria in the query, those products are pushed to the top of the ranking. The code uses filter scoring to ensure the featuredProduct will always appear above the alternateProduct.

import algoliasearch from 'algoliasearch';

const featuredProduct = '41469303161004';
const alternateProduct = '41469346644140';

const client = algoliasearch('H2M6B61JEG', 'b1bdfc3258823bb4468815a664dce649');

// Standard replica
const index = client.initIndex('shopify_algolia_products_price_asc_standard');

// with params
index.search(query, {
    facetFilters: [[
        "product_type:HardGood",
        "price_range:75:100"
    ]],
    optionalFilters: [[
        `objectID:${featuredProduct}<score=500>`,
        `objectID:${alternateProduct}<score=200>`
    ]],
    hitsPerPage: 50,
}).then(({ hits }) => {
    console.log(hits.map(item => `- ${item.title} | ${item.product_type} | ${item.objectID} | ${item.price_range}`).join('\n'));
});

Notice that this code uses a standard replica of the Shopify index sorted by price. Optional filters don't play well with Virtual Replica indices since both re-rank records at query-time, leading to unpredictable results. If you plan to use optional filters, you should use a standard replica that applies ranking at index-time.

You can also use negative optional filters to push records down the ranking. For example, if you wanted to push posts written by the current user lower in the rankings but not remove them completely:

index.search('', {
    filters: `date_timestamp > ${Math.floor(d.setDate(d.getDate() - 7) / 1000)}`,
    optionalFilters: [
      `author:-${user.name}`
    ],
    hitsPerPage: 50,
}).then(({ hits }) => {
    console.log(hits};
});

Optional filters are a powerful tool to add to your ranking tool belt, but remember that any query-time calculations will impact search performance. For instance, you shouldn’t use filter scoring on searches that may return more than 100,000 results. Always try to move ranking criteria to index configuration when possible. Use optional filters only when you need to further tune your results at query-time using more real-time context.

Let me know if you find a great (or not so great) use case for optional filters in your search UI!

Add autocomplete search to your Strapi blog

Chuck Meyer — Tue, 05 Apr 2022 17:26:20 +0000

Strapi is an open-source, headless CMS that builds API abstractions to retrieve your content regardless of where it's stored. It's a great tool for building content-driven applications like blogs and other media sites.

In this post, you're going to add interactive search to a Strapi blog with a Next.js front end using Algolia's Autocomplete.js library and the community-built Search plugin for Strapi.

Prerequisites

Strapi and Algolia Autcomplete.js are both build using Javascript, so you'll want node v.12 installed.

You'll build on the basic blog application from this guide using Strapi and Next.js. You should familiarize yourself with the steps in that post before building your search experience on top of it.

You'll also need an Algolia account for search. You can use your existing Algolia account or sign up for a free account.

Building the back end

Start by creating a directory to hold your project's front and back ends:

mkdir blog-strapi-algolia && cd blog-strapi-algolia

Strapi has a set of pre-baked templates you can use to get a CMS up and running quickly. These all include Strapi itself with pre-defined content types and sample data. If you don't have a Strapi blog already, you can use the blog template to quickly set one up.

npx create-strapi-app backend  --quickstart --template @strapi/template-blog@1.0.0 blog

After the script finishes installing, add an admin user at http://localhost:1337 so you can log in to the Strapi dashboard. This script sets up most of back end for us, including a few demo blog posts. You can read more about everything that was setup in the quick start.

Next you need to index your demo content. Fortunately, the Strapi community has you covered with a Search plugin and Algolia indexing provider built by community member Mattias van den Belt. You can read more about Mattie's plugin in the documentation, but getting up and running only requires a couple of pieces of configuration.

Go ahead and stop your Strapi server so you can install the plugin using npm (or yarn).

cd backend && npm install @mattie-bundle/strapi-plugin-search @mattie-bundle/strapi-provider-search-algoliai --save

You'll need to add an Algolia API key and App ID to your Strapi environment. You can manage your keys by navigating the Algolia Dashboard under Settings > Team and Access > API Keys, or go directly to https://algolia.com/account/api-keys. Since Strapi is modifying your Algolia index, you'll need to provide either the admin API key (for demos) or create a key with appropriate access for your production project.

# .env
// ...
ALGOLIA_PROVIDER_APPLICATION_ID=XXXXXXXXXX
ALGOLIA_PROVIDER_ADMIN_API_KEY=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

With your credentials in place, the last step is to configure the plugin. Create or modify the ./config/plugins.js file in your backend directory. You want to tell the plugin to index the article and category content types for your blog.

'use strict';

module.exports = ({ env }) => ({
  // ...
  search: {
    enabled: true,
    config: {
      provider: 'algolia',
      providerOptions: {
        apiKey: env('ALGOLIA_PROVIDER_ADMIN_API_KEY'),
        applicationId: env('ALGOLIA_PROVIDER_APPLICATION_ID'),
      },
      contentTypes: [
        { name: 'api::article.article' },
        { name: 'api::category.category' },
      ],
    },
  },
});

Restart your Strapi server to pick up these environment variables and the new plugin (npm run develop). The Search plugin triggers when new content is Published, so you'll need to either Unpublish and Publish the demo articles, or create a new one. Load the Strapi admin panel (http://localhost:1337/admin) and navigate to Content Manager > COLLECTION TYPES > Article and either click on an existing article to Unpublish or click on Create new Entry. Click Publish on your article to index this entry into your Algolia application. You can do the same for the category content type if you'd like to index those as well.

Building the front end

Now that you've built your back end and populated your index, it's time to build the front end for you users.

Strapi has a great blog post walking you through building a Next.js powered front end. You'll build on top of those steps here. You can either walk through their quick start yourself, or you can just clone this repo if you want to jump directly to adding search.

git clone --single-branch --branch no-search-version git@github.com:chuckmeyer/blog-strapi-frontend.git frontend

Don't forget to run cd frontend && npm install if you cloned the front end from the repo.

This is enough to get the basic blog site up and running. You can test it out by running npm run dev in the frontend directory.

The only thing missing is search. You'll be using the Algolia Autocomplete.js library to add an autocomplete search experience to your blog's navigation bar. When a user types into the field, the autocomplete "completes" their thought by providing full terms or results. The Autocomplete library is source agnostic, so you'll also need the Algolia InstantSearch library to connect to your index on the back end.

npm install @algolia/autocomplete-js algoliasearch @algolia/autocomplete-theme-classic --save

To use the autocomplete library in a React project, you first need to create an Autocomplete component to wrap the library. You can find the boilerplate for this in the autocomplete documentation.

./frontend/components/autocomplete.js

import { autocomplete } from '@algolia/autocomplete-js';
import React, { createElement, Fragment, useEffect, useRef } from 'react';
import { render } from 'react-dom';

export function Autocomplete(props) {
  const containerRef = useRef(null);

  useEffect(() => {
    if (!containerRef.current) {
      return undefined;
    }

    const search = autocomplete({
      container: containerRef.current,
      renderer: { createElement, Fragment },
      render({ children }, root) {
        render(children, root);
      },
      ...props,
    });

    return () => {
      search.destroy();
    };    
  }, [props]);

  return <div ref={containerRef} />;
}

Just like in the back end, you'll need your Algolia credentials to connect to the API. Since the front end only needs to read from the index, you can use your search key for the application. Create a ./frontend/.env.local file to store your credentials.

NEXT_PUBLIC_ALGOLIA_APP_ID=XXXXXXXXXX
NEXT_PUBLIC_ALGOLIA_SEARCH_API_KEY=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

Now you can initialize your connection to Algolia and add your new Autocomplete component by updating the code in ./frontend/components/nav.js.

import React from "react";
import Link from "next/link";

import { getAlgoliaResults } from '@algolia/autocomplete-js';
import algoliasearch from 'algoliasearch';
import { Autocomplete } from './autocomplete';
import SearchItem from './searchItem';
import "@algolia/autocomplete-theme-classic";

const searchClient = algoliasearch(
  process.env.NEXT_PUBLIC_ALGOLIA_APP_ID,
  process.env.NEXT_PUBLIC_ALGOLIA_SEARCH_API_KEY,
);

const Nav = ({ categories }) => {
  return (
    <div>
      <nav className="uk-navbar-container" data-uk-navbar>
        <div className="uk-navbar-left">
          <ul className="uk-navbar-nav">
            <li>
              <Link href="/">
                <a>Strapi Blog</a>
              </Link>
            </li>
          </ul>
        </div>
        <div className="uk-navbar-center">
          <Autocomplete
            openOnFocus={false}
            detachedMediaQuery=''
            placeholder="Search for articles"
            getSources={({ query }) => [
              {
                sourceId: "articles",
                getItemUrl( {item} ) { return `/article/${item.slug}`},
                getItems() {
                  return getAlgoliaResults({
                    searchClient,
                    queries: [
                      {
                        indexName: "development_api::article.article",
                        query,
                      }
                    ]
                  })
                },
                templates: {
                  item({ item, components}) {
                    return <SearchItem hit={item} components={components} />;
                  }
                }
              },
            ]}
          />
        </div>
        <div className="uk-navbar-right">
          <ul className="uk-navbar-nav">
            {categories.map((category) => {
              return (
                <li key={category.id}>
                  <Link href={`/category/${category.attributes.slug}`}>
                    <a className="uk-link-reset">{category.attributes.name}</a>
                  </Link>
                </li>
              );
            })}
          </ul>
        </div>
      </nav>
    </div>
  );
};

export default Nav;

As you can see, you're passing a few parameters to the Autocomplete component:

openOnFocus={false} - tells your search not to populate results until a user starts typing
detachedMediaQuery=''- opens search in a detached modal, providing more room for your results
placeholder="Search for articles" - the text that appears in the searchbox before a search
getSources={({ query }) => - where you define your data sources for your autocomplete experience

Remember that Autocomplete is source agnostic. You define sources based on APIs, libraries, or static content within your application. Here, you're binding a source called articles to your Algolia index using the getAlgoliaResults function from the autocomplete-js library.

              {
                sourceId: "articles",
                getItemUrl( {item} ) { return `/article/${item.slug}`},
                getItems() {
                  return getAlgoliaResults({
                    searchClient,
                    queries: [
                      {
                        indexName: "development_api::article.article",
                        query,
                      }
                    ]
                  })
                },
                templates: {
                  item({ item, components}) {
                    return <SearchItem hit={item} components={components} />;
                  }
                }
              },

The development_api::article.article is the index generated by the Strapi Search plugin above for your article content type. When you move to production, the plugin will create a separate production_api::article.article index in the same application.

The getItemUrl() section sets up keyboard navigation, while getItems() handles retrieving articles from your index using the query term(s) from the searchbox.

Notice the code above references a SearchItem component. This is the template you'll use to tell Autocomplete how to render your search results. Add a new component called ./frontend/components/searchItem.js with the following code.

import React from "react";

function SearchItem({ hit, components }) {
  return (
    <a className="aa-ItemLink" href={`/article/${hit.slug}`}>
      <div className="aa-ItemContent">
        <div className="ItemCategory">{hit.category.name}</div>
        <div className="aa-ItemContentBody">
          <div className="aa-ItemContentTitle">
            <components.Highlight hit={hit} attribute="title" />
          </div>
          <div className="aa-ItemContentDescription">
            <components.Highlight hit={hit} attribute="description" />
          </div>

        </div>
      </div>
    </a>
  );
};

export default SearchItem;

With this code, you're displaying the category associated with the article, the title, and the description. Use the components.Highlight component to emphasize the part of the attribute that matched the user's query.

And with that, you're done! Start your front end server with npm run dev. You should now see the autocomplete searchbox at the top of the page. Clicking on it opens the modal search interface where you can start typing your search term.

You can see a hosted version of this front end on codesandbox, although it may take some time for the back end container to start. The before and after versions of the front end code are both available on Github as well.

If you build something cool using this blog, share it with us on Twitter (@algollia).

Adding OAuth2 authentication to an AWS S3 static bucket with Okta

Guillaume Truchot — Wed, 23 Mar 2022 10:40:28 +0000

Our team recently implemented an internal corporate static website that allows employees to download technical reports.

Since we're heavy AWS users, we naturally decided to host it on AWS S3, which provides a dedicated feature to build static websites (S3 static website hosting).

However, we quickly ran into an issue: AWS S3 does not provide any native, out-of-the-box authentication/authorization process. Because it was an internal-only website, we needed some kind of authorization mechanism to prevent non-authorized users from accessing our website and reports.

We needed to find a solution to secure our internal static website on AWS S3.

Discovering the solution with CloudFront and Lambda@Edge

We use Okta for all Identity and User Management, so whatever solution we found had to plug-in with Okta.

Okta has several authentication/authorization flows, all of which require the application to perform a back-end check, such as verifying that the response/token returned by Okta is legit.

So we needed to find a way to carry these checks/actions on a static website which uses a back end that we don't control. That's when we learned about AWS Lambda@Edge, which lets you run Lambda Functions at different stages of a request and response to and from CloudFront:

We can trigger a Lambda Function at four different stages:

When the request enters CloudFront (viewer-request)
When the request goes out to the origin (origin-request)
When the response is returned from the origin (origin-response)
When the response is returned from CloudFront (viewer-response)

We saw a solution to our original issue: trigger a Lambda at the viewer-request stage that would check if the user is authorized. Two conditions:

If the user is authorized, let the request continue and return the restricted content
If the user is not authorized, send an HTTP response to redirect them to a login page

Implementing the Lambda@Edge function

We'll cover here the key elements and main issues we faced. The complete code is available here. Feel free to use it in your project!

Lambda@Edge restrictions and caveats

During the development of the solution, we ran into several restrictions and caveats of Lambda@Edge.

1 – Environment variables

Lambda@Edge Functions cannot use environment variables. That meant that we needed to find another way to pass data to our function. We opted for SSM parameters and templated parameter names in the Node.js code (we use Terraform to render the template when deploying the Lambda Function).

2 – Lambda package size limit

For viewer events (reminder: we use the viewer-request event), the Lambda package can be 1 MB at most. One MB is pretty small considering that it includes all dependencies (except of course the runtime/standard library) of your Lambda Function.

That's why we had to rewrite our Lambda in Node.js instead of the original Python, because the Python package with its dependencies exceeded the 1 MB limit.

3 – Lambda region

Lambda@Edge functions can only be created in the us-east-1 region. It's not a big issue but it means you'll need to:

Provision your AWS resources in that region to make things easier
In Terraform, you'll need to have a separate AWS provider to access the bucket you want to protect if it's not in us-east-1

4 – Lambda role permission

The IAM execution role associated with the Lambda@Edge functions must allow the principal service edgelambda.amazonaws.com in addition to the usual lambda.amazonaws.com. See AWS - Setting IAM permissions and roles for Lambda@Edge.

Authorization mechanism with Okta

Once we managed the above restrictions and caveats, we focused on the authorization/authorization.

Okta offers several ways to authenticate and authorize users. We decided to go with OAuth2, the industry-standard protocol for authorization.

Note
Okta implements the OpenID Connect (OIDC) standard which adds a thin authentication layer on top of OAuth2 (that's the purpose of the ID token mentioned hereafter). Our solution would also work with pure OAuth2 with minimal modifications (removal of the ID token use in the code).

OAuth2 itself offers several authorization flows depending on the kind of application using it. In our case, we need the Authorization Code flow.

Here is the complete diagram of the Authorization Code flow taken from developer.okta.com that shows how it works:

Our Lambda Function redirects the user to Okta where they will be prompted to login
Okta redirects the user to our website/Lambda Function with a code
Our Lambda Function checks if the code is legit and exchanges it for access and ID tokens by sending a request to Okta
Depending on the result returned by Okta, we:
- Allow or deny access to the restricted content
- If access is allowed, save the access and ID tokens in a cookie to avoid having to re-authorize the user on every page

Using JSON Web Tokens to store authorization result

So far we have a working authorization process; however, we need to check the access/ID token on every request (a malicious user could forge an invalid cookie/tokens). Checking the tokens means sending a request to Okta and waiting for the response on every page the user visits, which slows down the loading times significantly and is clearly sub-optimal.

Note
While local verification of the Okta token is theoretically possible, as of this writing the SDK provided by Okta uses a LRU (in-memory) cache when fetching the keys used to check the tokens. Because we're using AWS Lambda, and memory/state of the program isn't kept between invocations, the SDK is useless to us: it would still send one HTTP request to Okta for every user request, to retrieve the JWKs (JSON Web Keys). Worse, there is a limitation of 10 JWK requests per minute, which would make our solution stop working if there is more than 10 requests per minute.

We decided to use JSON Web Tokens to work around this. The initial authorization process is the same except that, instead of saving the access/ID tokens into a cookie, we create a JWT containing these tokens, and then save the JWT into a cookie.

Since the JWT is cryptographically signed:

A malicious actor cannot forge one (they would need the private key used to sign them)
The checking step required on every request is fast: we traded a long and I/O expensive HTTP request for a quick cryptographic check.

Note on JWT expiration and renewal

The JWT has a pre-defined expiration time which should be reasonably short, to avoid having a valid JWT containing expired or revoked access/ID tokens. Another option would be to check the access/ID tokens regularly and revoke the associated JWT if needed, but then we would need a revocation mechanism, which would makes things more complex.

Finally, as suggested above, the tokens provided by Okta have an expiration time. It is possible to transparently renew them using a refresh token (so the user doesn't have to re-login when the tokens expire) but we didn't implement that.

Conclusion

While adding OAuth2 authentication to an S3 static bucket with Okta (or any other OAuth2 provider) is possible in an AWS-integrated and secure manner, it's certainly not straightforward.

It requires writing a middleware between AWS and the OAuth2 provider (Okta in our case) using Lambda@Edge. We had to do the following ourselves:

Validate the user authentication
Remember the user authentication
Refresh the user authentication (not implemented in our solution)
Revoke the user authentication (TTL is implemented, but revocation before the end of the TTL is not)

Finally, a bunch of AWS resources must be created to glue everything together and make it work.

But it's worth it, because it works and our website is now more secure.

You can find the code of the Lambda@Edge as well as the infrastructure (Terraform) here:
https://github.com/GuiTeK/aws-s3-oauth2-okta

Build a store locator that includes online sellers

Chuck Meyer — Tue, 22 Mar 2022 14:35:39 +0000

The problem

Let's say you've been tasked to build an application to help consumers find agencies providing a specific service. Some of these agencies are local brick-and-mortar storefronts and others are online-only agencies that service the same local area. This problem was initially posed by Alejo Arias in the Algolia Discourse forum:

I have a series of providers in my index:

some are national providers (they have a national: true attribute)

some are state-wide providers (they have a state attribute with the list of states they service, e.g. [“NY”, “FL”])

some are local (they have specific lat/lng in their _geoloc attribute)

I want in my results anything that matches local providers by location close to my users, but also provide (in the same results) state and national providers. Adding a aroundLatLng filter automatically removes other results, no matter what facets or filters I try.

How can I achieve this?
Basically I want to have something like: aroundLatLng: x,y OR state: NY OR national: true

So how do you combine results from a geographic search for brick-and-mortar stores with other results from boolean or text-based queries? And how do you build an interface to display them in a unified way?

Geographic data and Algolia Search

As Alejo mentions, you can use Algolia for geographic searches by adding a special _geoloc attribute on your records. You put one or more sets of latitude/longitude tuples into this attribute to indicate geographic locations linked to the record.

Then you use the Algolia client libraries to query against these geocoded records -- filtering either within a radius around a fixed point (aroundLatLong) or the area within a shape (insideBoundingBox or insidePolygon). The documentation goes into more detail about the differences between these methods. You can also read through these posts that walk you through building a pure-geographic store locator.

However, you cannot extract geographic and non-geographic results from the same query. If you are searching for proximity, records lacking _geoloc attributes will not show up in the result set.

So how do you perform this search when not all records have geographic coordinates?

A single index solution

You could do the whole thing via geographic search. By adding _geoloc data to the state and national records, you could search for everything using a geographic query. For instance, placing the statewide agencies at coordinates at the center of each state. This was the initial solution I added to the forum post, but there are several problems with this solution:

Alejo specifically mentions that some providers span multiple states
Placing providers at the center of the state will cause inaccurate results for consumers who live near state borders
National providers would need records in every state

A multi-index solution

As an alternative you can build a multi-index solution with one index for the brick-and-mortar storefronts that include geographic data, and another for the state and national providers. You can then search the two data sources independently and blend the result sets. This approach requires two Algolia queries per search, but it will allow us to guarantee results from both types of providers.

Preparing your indices

First, you'll need an agency data set. You can build one from scratch using a couple of sources. You can start with anonymized address data from this repo containing about 3000 addresses across the United State. Then, run these addresses through a small script to add fictional agency names and randomly flag some of the agencies as "preferred".

def transform_records(addresses):
  address_records = []
  for address in addresses:
    record = {}
    record_geocode = {}
    # One in ten chance agency is preferred 
    record['preferred'] = 10 == random.randint(1,10)

    record['objectID'] = random_name.generate_name().title()
    if record['preferred']:
      record['name'] = f"{record['objectID']} Agency (Preferred)"
    else:
      record['name'] = f"{record['objectID']} Agency"
    record['address'] = address.get('address1')
    record['city'] = address.get('city')
    record['state'] = address.get('state')
    record['zip'] = address.get('postalCode')
    record_geocode['lat'] = address['coordinates']['lat']
    record_geocode['lng'] = address['coordinates']['lng']
    record['_geoloc'] = record_geocode
    address_records.append(record)
  return address_records

You can use another script to generate statewide and multi-state agencies for the second index. Both data sets reside in this repo. You can create indices from these data sets under your existing Algolia account or sign up for a free account and set up a new agency_finder application.

Building the front end

Now that you've populated your indices, it's time to build the front end. Algolia's geoSearch component in the InstantSearch library includes a helper component to initialize the Google Maps API, render a Map, and tie that map to geolocation queries in your Algolia index. It's the same component I used previously to build a COVID-19 case visualizer. However, for this project, you want the user to type in an address and resolve the geolocation information for them using the Google Places API. This proves challenging using the out-of-the-box components in InstantSearch, so you'll build your own interface from scratch.

This blog post gives us a solid model for building an address autocomplete form in React. You'll use this as the foundation for your AgencyFinderForm component to render the address autocomplete input field as well as read-only fields to display the resulting address. The latitude/longitude are stored in state, but not shown on the form

You can modernize the code from the blog by using the Google Wrapper around your React components to initialize the google object and add the Places API.

   renderForm = (status) => {
    switch (status) {
      case Status.SUCCESS:
        return <AgencyFinderForm handleCallback={this.handleCallback} />;
      default:
        return <h3>{status} ...</h3>;
      };
  }

  render() {
    return (
      <div>
        <h1>Find an Agency</h1>
        <p className='instructions'>🔍 Search for your address to find the closest agencies.</p>
        <div className='left-panel'>
          <Wrapper apiKey={process.env.REACT_APP_GOOGLE_API_KEY} render={this.renderForm} libraries={["places"]} />
        </div>
        <div className='right-panel'>
          <AgencyFinderResults hits={this.state.results} />
        </div>
      </div>
    )
  }
}

Next you add a clear button to the basic form.

  handleClear() {
    this.setState(this.initialState);
    var input = document.getElementById('autocomplete');
    input.value = '';
    google.maps.event.removeListener(this.autocompleteListener);
    this.initAutocomplete();
  }

Finally, you'll clean up processing the address_components from the Places API with following code:

  handlePlaceSelect() {
    const addressObject = this.autocomplete.getPlace();
    const address = addressObject.address_components.reduce((seed, { short_name, types }) => {
      types.forEach(t => {
        seed[t] = short_name;
      });
      return seed;
    }, {});
    [this setState](this.setState)({
      streetAddress: `${address.street_number} ${address.route}`,
      city: address.locality ? address.locality : address.sublocality_level_1,
      state: address.administrative_area_level_1,
      zipCode: address.postal_code,
      geoCode: addressObject.geometry.location.lat() + ', ' + addressObject.geometry.location.lng(),
    });
  }

Querying for results

After the user has selected a location and you have the latitude, longitude, and address information stored in the component state, you're ready to query the indices. You use the multipleQueries method from the Javascript API client to batch together the two queries and combine the results. This will still count as two queries against your Algolia limit, but it reduces the number of round trips to the API.

handleSubmit(event) {
    const queries = [{
      indexName: statesIndex,
      query: this.state.state,
      params: {
        hitsPerPage: 10
      }
    }, {
      indexName: geoIndex,
      query: '',
      params: {
        aroundLatLng: this.state.geoCode,
        facetFilters: [ this.state.preferred ? 'preferred:true' : '' ],
        hitsPerPage: 10,
      }
    }];

    this.searchClient.multipleQueries(queries).then(({ results }) => {
      let allHits = [];
      results.map((result) => {
        return allHits.push(...result.hits);
      });
      this.props.handleCallback(allHits);
    });
  }

First, you initialize the two queries. Notice how the multipleQueries method allows us to mix geographic and string-based queries, and even layer in an
optional facetFilter for your "Preferred" agencies. You then pass the array of queries to the client. The response includes the individual results from each
query, but you can just smash the hits from the two result sets into a single array and pass them to the AgencyFinderResults component.

Putting it all together

You now have a solid proof-of-concept React component for layering geographic and non-geographic results into a single result set. At this point you could improve the example by adding a Google Map to display the geographic results. You could also pivot back to a single index, using multipleQueries ability to query the same index multiple times with different parameters.

The complete example is available in this Github repo or you can try out a live demo.

Centralizing state and data handling with React Hooks: on the road to reusable components

Emmanuel Krebs — Thu, 10 Mar 2022 14:31:33 +0000

Application development is often reactive. We see the need, we deliver a solution as fast as possible. During this fast software cycle, we gather requirements and implement them as soon as they appear. I’m not talking about quick and dirty. I’m referring to using the best RAD practices - rapid application development.

The RAD cycle is as follows: you implement great core features (MVP-style), relying on years of experience to create maintainable code. But over time, several things occur: requirements change, more code gets written, and the codebase starts to rebel against your intuitively brilliant but perhaps not fully robust architecture. So you start refactoring. Also, you discover that technology changes, offering new ways to make your code simpler, cleaner, and more powerful.

Enter game changer React Hooks. And, a fast growing business that requires you to rewrite your application with loads of new features.

Rewrite – from scratch. Life offers a second opportunity.

How React Hooks saved our administration application

Application development can also be pro(Re)active. Our administration application is data-intensive. Previously, many separate (and competing) components had managed their data independently - connecting, formatting, displaying, updating, etc..

The requirements of an Admin application

An Admin application is a good candidate for centralizing data handling. Administrators need to see the data as is, so the onscreen views usually match the structure of the underlying data. So, while our client-facing dashboard presents functional views for business users, an administrator needs to see user or client subscription information in a consistent and straightforward manner.

What we needed was a more scalable solution. Since we pull data from multiple sources – all accessible via one API with many endpoints – we wanted to centralize the common aspects of data handling. This not only gave us immediate benefits (better testing, caching, syncing, standard typing), it facilitated and simplified future data integrations.

A customized hook

We implemented a custom React hook called useData, which manages and therefore centralizes all data-retrieval API calls, data exchanges, type checking, caching, and other such data-based functionality. The caching alone enhanced user-facing speed enormously. Equally important, the speed and centralization enabled our front-end developers to reuse their components and UI elements in different parts of the interface. Such reusability created a feature-rich, user-friendly UI/UX without front-end developers needing to maintain unique state information within each component. Lastly, under the hood, data reusability enabled a coherence in the models that drove the front-end functionality. We will discuss front-end benefits of React hooks in future articles; this article is about how we served the front-end with a reliable and scalable layer of data handling.

How our `useData`hook centralized the process

We use different data sources, some more complex than others but all following the same JsonAPI specification. Additionally, they all have the same needs – a means to:

Retrieve data
Deserialize and format it
Validate its format
Perform error handling (data quality, network)
Synchronize with app refreshes and other data/workflows
Cache the data and keep it up to date Enough talking, here is our useData hook code:

import { useCallback } from 'react';
import { useQuery, useQueryClient } from 'react-query';
import { ZodObject, infer as Infer } from 'zod';
import { useApi } from 'hooks';
import { metaBuilder, MetaInstance } from 'models';

interface Options {
  forceCallApi?: boolean;
  preventGetData?: boolean;
}

interface ApiData<T> {
  data?: T;
  meta?: MetaInstance;
}

export interface DataResult<Output> {
  data?: Output;
  meta: any;
  loading: boolean;
  errors: Error[];
  refresh: () => Promise<void>;
}

export const useData = <Model extends ZodObject<any>, ModelType = Infer<Model>, Output extends ModelType = ModelType>(
  builder: (data: ModelType) => Output,
  url: string,
  { forceCallApi = false, preventGetData = false }: Options = {}
): DataResult<Output> => {
  const queryClient = useQueryClient();

  const { getData } = useApi(url);

  const getDataFromApi = useCallback(async (): Promise<ApiData<Output>> => {
    // here we get the data (and meta) using getData, and handle errors and various states
    return { data: builder(apiData), meta: metaBuilder(apiMeta) }
  }, [getData, builder, queryClient, url, forceCallApi]);

  const { data: getDataResult, isLoading, error } = useQuery<ApiData<Output>, Error>(
    [url, forceCallApi],
    getDataFromApi,
    { enabled: !preventGetData, cacheTime: forceCallApi ? 0 : Infinity }
  );

  const refresh = useCallback(async () => {
    await queryClient.refetchQueries([url, forceCallApi], {
      exact: true,
    });
  }, [queryClient, url, forceCallApi]);

  return {
    data: getDataResult?.data,
    meta: getDataResult?.meta,
    loading: isLoading,
    errors: ([error]).filter((error) => error !== null) as Error[],
    refresh,
  };
};

As you can see, this hook takes three parameters that, when combined, give us all the following functionalities:

A “builder” function that transforms and enhances the data for use by our components
The URL of the API endpoint that retrieves the data
Optional parameters. For example, to ignore cache or wait for some other data to be ready before calling the API The result is that our components no longer need to manage all that. We’ve abstracted and encapsulated the complexity. The useData hook returns some values we can use in our components:
Some states: loading and errors (if any)
The data (if any)
Meta information (if present – pagination information, for example)
A refresh function (to refresh the data by calling the API again) ## Building the data Let's take a deeper look at what this code does and how we use it. ## Schema validation with Zod Getting the data is one thing. Ensuring that the data is correctly structured, or typed, is another. Complex data types require validation tools like yup or zod that enforce efficient and clean methods, and offer tools and error handling runtime errors based on faulty types. Our front end relies on strongly-typed data sets, so the validation stage is crucial for us. We use zod. Zod is used to build a model of the data. For example, here is what the model for our Application could look like:

import { object, string, number } from 'zod';

const Application = object({
  applicationId: string(),
  name: string(),
  ownerEmail: string(),
  planVersion: number(),
  planName: string(),
});

Then, to construct our builder function, we use in-house-built generic helpers on top of the zod model.This helper takes two parameters:

The model of our data (Application in our example above)
A transformer function that is used to enrich that model. In our case, that transformer would look like this:

import { infer as Infer } from 'zod';

const transformer = (application: Infer<typeof Application>) => ({
  ...application,
  get plan() {
    return `${application.planName} v${application.planVersion}`;
  },
});

Another example of enrichment is if a model has a date: we usually want it to expose a javascript date rather than a string date.

We have 2 versions of that helper function (one for objects and one for arrays). Below is the first one:

import type { ZodType, TypeOf, infer as Infer } from 'zod';
import { SentryClient } from 'utils/sentry';

export const buildObjectModel = <
  Model extends ZodType<any>,
  ModelType = Infer<Model>,
  Output extends ModelType = ModelType
>(
  model: Model,
  transformer: (data: TypeOf<Model>) => Output
): ((data: ModelType) => Output) => {
  return (data: ModelType) => {
    const validation = model.safeParse(data);
    if (!validation.success) {
      SentryClient.sendError(validation.error, { extra: { data } });
      console.error('zod error:', validation.error, 'data object is:', data);
      return transformer(data);
    }
    return transformer(validation.data);
  };
};

The typed output by zod is very clean and looks like a typescript type that we would have written ourselves, with the addition that zod parses the JSON using our model. For safety, we use the safeParse method from zod, which allows us to send back the JSON “as is” in case of an error during the parsing step. We would also receive an error on our error tracking tool, Sentry.

With our example, our builder function would look like:

export const applicationBuilder = buildObjectModel(Application, transformer);
// and for the record, here is how to get the type output by this builder:
export type ApplicationModel = ReturnType<typeof applicationBuilder>;
// which looks like this in your code editor:
// type ApplicationModel = {
//   plan: string;
//   applicationId: string;
//   name: string;
//   ownerEmail: string;
//   planVersion: number;
//   planName: string;
// }

Calling the API

Internally, we use another custom hook useApi (less than 200 lines of code) to handle the GET/POST/PATCH/DELETE. In this hook, we use axios to call the backend API and perform all typical CRUD functionality. For example, on the read side, Axios deserializes the data we receive before it is converted from the JSON API spec to a more classic JSON, and switching from snake_case to camelCase. It also handles any meta information we receive.

Also, from a process point of view, it manages request canceling and errors when calling the API.

Caching the data

At this point, we can summarize: the useApi hook gets the data, which is then passed through the builder to be validated and enriched; and the resulting data is cached using react-query.

We implemented react-query for caching the data on the front end, using the API endpoint URL as the cache key. React-query uses the useApi hook mentioned above to fetch, synchronize, update, and cache remote data, allowing us to leverage all these functionalities with a very small codebase.
All we have to do on top of that is to implement react-query’s provider. To do so, we’ve constructed a small react component:

import { FC } from 'react';
import { QueryClient, QueryClientProvider, QueryClientProviderProps } from 'react-query';

export const queryClient = new QueryClient({
  defaultOptions: {
    queries: {
      refetchOnWindowFocus: false,
      refetchInterval: false,
      refetchIntervalInBackground: false,
      refetchOnMount: false,
      refetchOnReconnect: false,
      retry: false,
    },
  },
});

type IProps = Omit<QueryClientProviderProps, 'client'> & {
  client?: QueryClient;
};

export const GlobalContextProvider: FC<IProps> = ({
  children,
  client = queryClient,
  ...props
}) => (
  <QueryClientProvider {...props} client={client}>
    {children}
  </QueryClientProvider>
);

Most importantly, it manages our caching. We have many components that need the same data, so we wanted to avoid unnecessary network traffic to retrieve the same information. Performance is always key. So is limiting potential errors performing unnecessary network calls. Now, with caching, if one component asks for data, our cache will store that data and give it to other components that ask for the same information. In the background, React-query of course ensures that the data in the cache is kept up to date.

To sum up, here is an example of a component built using this useData hook and our Application model as defined above:

import { FC } from 'react';

interface ApplicationProps {
  applicationId: string;
}

export const ApplicationCard: FC<ApplicationProps> = ({ applicationId }) => {
  const { loading, data: application, errors } = useData(applicationBuilder, `/applications/${applicationId}`);

  return loading ? (
    <div>loading...</div>
  ) : errors.length > 0 ? (
    <div>{errors.map(error => (<div>{error}</div>))}</div>
  ) : (
    <div>
      <div>{application.applicationId}</div>
      <div>{application.ownerEmail}</div>
      <div>{application.name}</div>
      <div>{application.plan}</div>
    </div>
  );
};

As you can see, our useData hook lets us standardize the loading & errors states, thus encouraging us to write reusable components that handle those states. For example, we have reusable StateCard and StateContainer components. With the data now easily available, we can go about integrating those reusable components and focus exclusively on building a great front end experience – cleanly, fully-featured, and scalable.

API keys vs JWT authorization – Which is best?

Julien Bourdeau — Mon, 28 Feb 2022 10:18:29 +0000

As you build your own APIs, examining your use cases will help you decide which security methods to implement for each API. For some use cases, API keys are sufficient; in others, you’ll want the additional protection and flexibility that comes with JSON Web Tokens (JWT) authorization.

So in the comparison API keys versus JWT authorizations, the winner is .. it depends.

All API calls require some measure of security and access control. API keys with a sensible ACL can provide enough security without adding too much overhead. But with the increased use of microservices for nearly every small and large task, your API ecosystem may need a more unified, granular, and secure method like JWT authorization.

When API keys are fine

Online businesses that use a cloud-based Search API can normally expose read-only API keys without great risk – if the underlying index of data contains no secrets. In fact, client-side apps should connect to the cloud search engine directly for performance reasons – thereby exposing their API key – to avoid the lengthier trip to the back-end server before going to the cloud. On the other hand, index updates require restricted access API keys, which should never be exposed.

But in both use cases (search and indexing), API keys are generally fine; there’s no urgent need for the overhead of JWT authorization.

When it’s time to consider JWT authorization

Increasingly, however, APIs demand more flexibility and protection. JWT authorization not only adds an additional level of security (more about this below), it also provides a more manageable and easier method to orchestrate the multitude and network of APIs used on a daily basis. As described in the next sections, JWT centralizes authentication & authorization by generating a single shared token that contains user and app-level information (encrypted or hashed) to help any API in the same ecosystem determine what the token-holder is allowed to do.

API keys, at first glance, seem so simple – you just need to send the proper API key and you’re ready to go. But that’s a bit deceptive. When your ecosystem relies on many integrated microservices, managing numerous API keys becomes messy, unreliable, and nearly impossible to manage. They grow in number, they change, they expire, they get deleted, their ACLs change – all that and more, without notifying the apps and users relying on these same API keys.

With JWT, you lay the foundation for a single sign-on architecture. We discuss this below in the section Switching over to JWT.

Using API keys vs JWT authorization

Using Api Keys

API keys are direct, simple, and fully transparent. They don’t represent any underlying information, they do not encrypt a secret message. They are simply an unreadable unique id.

Here’s an example of a publicly available API key in client-side javascript. The code includes an App ID (app-id-BBRSSHR\) that uses the API key (temp-search-key-ere452sdaz56qsjh565d\) to allow it to search. The App ID refers to one of your user-facing apps (like an online website or streaming service). The API key is temporary and short-lived (expiring after some period of time) to provide some protection from unwanted use or abuse.

import { hitTemplate } from "./helpers";
const search = instantsearch({
  appId: "app-id-BBRSSHR",
  apiKey: "temp-search-key-ere452sdaz56qsjh565d",
  indexName: "demo\_ecommerce"
});

Another example: Indexing, which requires a more secure API key. It has the same format (appId + apiKey), but it is private because it is hidden from the public, either in compiled code or a secure database on your back end. The App ID (YourApplicationID) refers to the back office system. The API key (YourAdminAPIKey) might be a permanent admin key that changes only once a year for simpler maintenance.

use Algolia\\AlgoliaSearch\\SearchClient;
$client = SearchClient::create(
    'YourApplicationID',
    'YourAdminAPIKey'
);
$index = $client->initIndex('demo\_ecommerce');
$index->saveObject(
  \[
    'firstname' => 'Jimmie',
    'lastname'  => 'Barninger',
    'city'      => 'New York',
    'objectID'  => 'myID'
  \]
);

Using the JWT token

A JWT token is a large unreadable set of characters that contains hidden and encoded information, masked by a signature or encryption algorithm. It’s made up of three parts: a header, body, and signature. They are separated by a period: Header.Body.Signature.

EZPZAAdsqfqfzeezarEUARLEA.sqfdqsTIYfddhtreujhgGSFJ.fkdlsqEgfdsgkerGAFSLEvdslmgIegeVDEzefsqd

The JWT header is EZPZAAdsqfqfzeezarEUARLEA, which contains the following information:

{
  "alg": "HS256",
  "typ": "JWT"
}
{
  "sub": "1234567890",
  "name": "John Doe",
  "iat": 1516239022
}

There are different algorithms available, for example RS256 and HS256. Here we use HS256 which requires a private key to be used when generating the signature. RS256\ uses both a private and a public key combination.

The JWT body (called the payload) is sqfdqsTIYfddhtreujhgGSFJ, which contains the user’s identity to help establish the token user’s rights. It also gives other information, such as an expiration date (the shorter the more secure):

{
  "sub": "1234567890",
  "name": "John Doe",
  "iat": 1516239022
}

The signature is fkdlsqEgfdsgkerGAFSLEvdslmgIegeVDEzefsqd, which is generated by combining the header, body, and a shared private key, using the HS256 hashing method, as indicated in the header.

HMACSHA256(
  base64UrlEncode(header) + "." +
  base64UrlEncode(payload),
  secret)

And so that’s how you get the following token: Header.Body.Signature:

EZPZAAdsqfqfzeezarEUARLEA.sqfdqsTIYfddhtreujhgGSFJ.fkdlsqEgfdsgkerGAFSLEvdslmgIegeVDEzefsqd

A word about authentication & authorization

Both API key and JWT are used for authentication and authorization, but they do it differently.

Authentication allows the user or application to use one or more methods of the API.
Authorization defines how they can use those methods. Some apps or users can only read the data; others can update; others are administrators (roles and permissions). Same goes for API keys, as managed by their ACLS - they can be read-only, write-access, or admin.

API keys authenticate and authorize using the same API key. JWT Authorization requires an initial authentication process before it generates the authorization token. Once the token is generated, it is used across the ecosystem to determine what the token holder can and cannot do.

Additionally, API keys authenticate the application not the user; whereas, JWT authenticates both the user and the application. Of course, you could use API keys for user-level authorization, but it’s not well-designed for that - an ecosystem would need to generate and manage API keys for every user or session id, which is unnecessarily burdensome for a system.

A word about better protection and security

In terms of security, both API keys and JWT are open to attacks_._ The best security measure is to implement a secure architecture for all end-to-end communications.

That said, API keys are historically less secure because they rely on being hidden. You can hide keys with SSL/TLS/HTTPS, or by restricting their usage to back-end processes. However, you can’t control all API use; API keys are likely to leak; HTTPS is not always possible; and so on. With JWT, because the token is hashed / encrypted, it comes with a more secure methodology that is less likely to be exposed.

What information is in a JWT token?

The most notable difference between an API key and a JWT token is that JWT tokens are self-contained: they contain information an API needs to secure the transaction and determine the granularity of the token-holder’s rights. In contrast, API keys use their uniqueness to gain initial access; but then the API needs to find a key’s associated ACL in a central table to determine exactly what the key gives access to. Typically, the API key provides only application-level security, giving every user the same access; whereas the JWT token provides user-level access.

A JWT token can contain information like its expiration date and a user identifier to determine the rights of the user across the entire ecosystem.

Let’s take a look at some of the information you can include in a JWT token:

iss (issuer): identifies the principal that issued the JWT.
sub (subject): identifies the principal that is the subject of the JWT. Must be unique
aud (audience): identifies the recipients that the JWT is intended for (array of strings/uri)
exp (expiration time): identifies the expiration time (UTC Unix) after which you must no longer accept this token. It should be after the issued-at time.
nbf(not before): identifies the UTC Unix time before which the JWT must not be accepted
iat (issued at): identifies the UTC Unix time at which the JWT was issued
jti (JWT ID): provides a unique identifier for the JWT.

Example

{
    "iss": "stackoverflow", 
    "sub": "joe", 
    "aud": \["all"\], 
    "iat": 1300819370, 
    "exp": 1300819380, 
    "jti": "3F2504E0-4F89-11D3-9A0C-0305E82C3301",
    "context": 
    { 
        "user": 
        { 
            "key": "joe", 
            "displayName": "Joe Smith" 
        }, 
        "roles":\["admin","finaluser"\] 
    } 
}

JWT authorization offers flexibility, reliability, and more security

So here’s the scenario. You have many applications:

Apps that allow us to track the API usage of our all our users
Apps that give access to billing and customer data
Apps that allow API users to change settings on different systems
Apps that retrieve product data or business content
And other such use cases

Doing it with API keys

The problem arises when there are more APIs running the show. Where do you store the 100s+ of API keys needed for all this access? Managing too many API keys requires a table of API keys to be made available to all apps that run within the ecosystem.

Thus, every app in the ecosystem would have to be aware of the database. Each app would need to connect and read from that table. And some apps would be allowed to generate new keys or modify existing keys. All of which is fine, but becomes difficult to maintain. One way to manage this is to create an API that validates a key against this database. But for that, you’ll need a second authentication system to connect to this API.

Not only is retrieving the API keys cumbersome, but maintaining their duration and level of authorization becomes tedious. And not every app functions at the app level, they need user-level rights. Additionally, APIs can be used in ways that differ from system to system. Worse, different apps share API keys, and are therefore dependent on the different apps to maintain correct access-levels of the shared API keys.

Switching over to JWT

Any API that requires authentication can easily switch over to JWT’s authorization. With JWT authorization, you get a user-based authentication. Once the user is authenticated, the user gets a secure token that they can use on all systems. The management of the user (and therefore the token) is centralized. You set up access rights and you give each user different rights for each system. The JWT authorization endpoint authenticates the user and creates the token.

With that architecture, the next step is to create a single sign on into the full ecosystem and rely only on the token for rights. In the end, the simplest and most robust and manageable approach is to create this single endpoint dedicated to do both authentication & authorization, such that all other servers across the entire ecosystem can rely on that central point to authorize the API interactions between client and server.

In sum, sometimes JWT is absolutely needed and sometimes it’s overkill

As an application developer, my primary concern when building APIs is that they are used properly and provide the correct data and functionality. I therefore rely heavily on the advice of DevOps to suggest the best and most manageable security. But it’s not just about security. In a large ecosystem, we need a simple and robust way to access microservices across multiple systems and servers, which is best served by centralizing the authentication and authorization processes.

JWT is overkill in the following two situations:

A simple ecosystem with only a few APIs
Suppliers of third-party APIs.

Third-party APIs should be easy to use and fast to implement, with little upfront work when integrating. Additionally, you need to share the key used to sign the token. So it's best when you control both ends, which is unlikely in third-party sceneries. You also need to trust the implementation. for example an API could choose to ignore the expiration date or the NBF (“not before”).

JWT is not overkill when …

You’ll want to use JWT when multiple services need to communicate with each other over a vast network. Centralizing and securing those exchanges is crucial. It’s especially important when each network or app requires different levels of access based on the user, not just the app. It’s also important to have control over the traffic and to be able to prioritize network calls. Finally, you want the simple plug-and-play experience when adding new microservices or improving existing ones.

Integrating custom dictionaries into NLP for greater clarity

Joris Valette — Fri, 18 Feb 2022 09:07:41 +0000

Language is a funny thing. For example, we take for granted that Cinderella wore glass slippers. Only in a fairy tale can people walk in glass shoes. But maybe it’s metaphorical – the fragility of being Cinderella. Or maybe it’s simply a mistranslation of the original Latin word “vair” (squirrel fur) for the French “verre” (glass).

Language is also hard to pin down. Especially when lost in translation. But no need to despair – sometimes what gets lost can have surprising and moving results.

But we don’t always want to be surprised. Like when we ask direct questions or search for specific items that match our queries. At that point, we aspire to be crystal-clear.

That’s where dictionaries come in. Dictionaries allow us to be clear, by reinforcing the clarity of each word within the context of the larger phrase. We use custom dictionaries to reinforce our natural language processing (NLP). Here’s how.

Stop words and plurals, and compounds and segments – a few of my favorite things

Many users still type questions like “What is the best search engine?” instead of the shorter “best search engine”. It’s natural for them to type the way they speak. But other people prefer to use shorter, incomplete phrases to return the same results. With advances in search technology, even nonsensical queries, such as “engine best search”, return great results.

Nevertheless, the full phrase is still in fashion – even more so with voice. The success of voice search depends on allowing people to speak naturally. And stop words are key to that. Stop words reduce a natural phrase to its bare essence: keywords. By dropping such words as “what”, “is”, and “the” from the above query, and leaving only the keywords “best”, “search”, and “engine”, the search engine can match the query to the underlying data in a more reliable and relevant way.

Granted, all words are important – “What” and “Why” are indeed meaningful distinctions – but if a search algorithm relies on textual matching (as opposed to matching on meaning or semantics), its only job is to compare characters and words. By removing stop words, therefore, you remove the false positives that match on the word “the”.

We can say the same about normalization (e.g., removing accents), plurals. Any search algorithm that focuses on text, and not the meaning of text, should ignore textual variations (like plurals) to enable a more relevant and non-ambiguous word matching.

Lastly, textual matching also needs to separate words into useful parts. A “boat house” is not a house or a boat but a boat specially made to be used as a house. To help reach that level of precision, a textual search algorithm needs to break out the constituent parts of a word (atoms), by using techniques like segmentation and decompounding.

The goal of segmentation or decompounding is not to understand the meaning of words, but to find out what a complex word can be decomposed into. We’re trying to find the “atoms” of the word. We don’t use it in English because most words are already decompounded, it’s in the language’s DNA. Same for French. But German, for example, Hundehütte, meaning "dog kennel", is composed of “Hund” (dog) and “Hütte”‘ (kennel/house). The space we already have between the two words in English is why we don’t need decompounding. Segmentation is essentially the same thing, but for languages where there’s no space at all (i.e., most Asian languages).

That’s where dictionaries come in.

Using dictionaries for text-based matching

One approach to natural language processing is to use dictionaries, such as a stop-word dictionary, plurals dictionary, and a compound-word dictionary. For example, you can parse a downloaded list of stop words from Wiktionary, not only in English, but in many other languages.

Here’s the process we used:

Download the full wiktionary dictionary – words, definitions, and much more
Extract the words
Store them in text files
Compile them into a binary format
Optimize code for performance We do this for every language and it works fairly well for most use cases. But when it doesn’t work, it breaks relevance – which is a critical show-stopper for search engines. Here are some problems we encountered

I’m “down” with relevance

“Down” is a reasonable stop word, except when your searching for “down jackets”. Companies who sell “leather”, “suede”, and “down” jackets cannot remove “down” from the query.

Ambiguity cloaked in “fur” (not squirrel fur)

Languages that use accents, like French and Spanish, fare well when normalized with accent removal. For example, “voilà” to “voila” causes no loss in meaning. In fact, it’s rare in French that removing an accent would create an ambiguity. German is not so lucky. For example, the accented “ä”, when normalized to “a”, will change the meaning of some words.

A curious example of this is the German word “wählen”, meaning “to choose” in English. If you remove the accent, most people will not object – except for the 1500 residents of the small German-speaking Swiss town, Wahlen. It might be hard to find the town “Wahlen” among the many results that match on “wählen” – thus, hurting tourism in that part of the world.

The solution is to do a special custom normalization for german. In this case, normalize “ä” to “ae”. Here’s a complete list:

ä → ae
ö → oe
ü → ue
Ä → Ae
Ö → Oe
Ü → Ue
ß → ss (or SZ for capital)

But this leads to a second problem, which illustrates the gymnastics search engines go through when dealing with languages. (Remember, language is funny…) So we normalize “für” to “fuer”, but now we lose the stop word “fur”, because the now normalized “fuer” is not a stop word.

That’s where custom dictionaries come in.

The solution – Giving customers control with custom dictionaries

We realized that one dictionary per category wasn’t enough, we needed to come up with an additional dictionary per customer that they could use to override the defaults of Wiktionary or add their own words. So now we have two dictionaries per category (stop words, plurals, etc.): one per language, which we ship out with our software, and one custom dictionary per customer, which they can add words to. Adding custom dictionaries – meaning, allowing each customer to override and add their own words to our dictionaries – required a bit of refactoring in how we dealt with our standard dictionaries: each dictionary-retrieval function had a different interface and each dictionary dataset had different formats. So the first step was to normalize our code and data.

Normalizing our codebase and standardizing our dictionaries interface

We examined the current dictionaries that we shipped out to our customers as part of our base product. We wanted to abstract the similarity of every dictionary. Since they all had the same kind of data and goals, we were able to do the following:

Create similar data = a list of words
Code the same goal = the ability to retrieve words To put these dictionaries into a single interface, the main tasks included (in this order):
Transforming all the dictionary datasets to have the same data structure: the trie
Migrating existing dictionaries to the new format In the end, our new format for the plurals dictionary file is:

[2-letter country code]=[word1,word2,..]

In keeping with our introduction, here’s a good example of plurals:

en=feet,feets,foot,foots
en=slipper,slippers
en=squirrel,squirrels
en=fur,furs
en=Cinderella,Cinderellas
en=Cinderfella,Cinderfellas

That’s the first part: unifying both the interface and structure of the data.

With that, we achieved the following goals:

A simpler dictionary interface for all dictionaries
Mutualized toolings and tests
Easier to maintain

Plugging in customer-created, custom dictionaries

Now that we had a single interface for every dictionary, we were able to integrate customer-defined words for every NLP technique, for example, customer-specific stop words (see “down” example above), customer-specific normalization (see “für” example above), and so on.

These custom dictionaries are added to the index on top of the static dictionaries. We prioritized the dictionary lookups: a query first consults the custom dictionary before the static one. If the word is found, then the engine doesn’t need to look at the static dictionary.

And that’s it: our customers can now slip on one slipper and help their own customers find the other slipper(s) – in fur or glass.

Pest: a PHP testing framework that goes above and beyond PHPUnit

Peter Villani — Fri, 18 Feb 2022 08:56:54 +0000

When building third-party APIs, it’s important to test your code in every runtime scenario you’ve seen or can foresee. For this, a robust and easy to use & maintain testing framework is a must. As PHP developers, we relied heavily on PHPUnit, but we switched over to the Pest Testing Framework, which simplified and reduced our large library of testing code.

Given the number of tests we perform, and trying to cover every function in our codebase, we needed to simplify the testing code and make it easier to maintain, understand, and debug. We also wanted to make sure our testing framework would require little effort to integrate into Laravel.

In this article, we take a look at Pest, a clean & robust testing framework built on top of PHP’s standard testing library PHPUnit.

Reducing the testing codebase of PHPUnit with PHP chaining

Inspired in part by the one-line code “it” syntax in Ruby Rspec, Pest’s own syntax is fairly straightforward. Pest removes the namespace and library references needed in PHPUnit, and it doesn’t require extending hundreds of functions. It simply needs you to specify the test function – making it far easier to maintain and debug:

it('is an example, function () {
    assertTrue(true);
});

Still, 3 lines of code might be too much when multiplied by 100s of tests. Pest allows you to chain the functions. The result is a one liner:

it('is an example', function ()->assertTrue(true);

Thus, Pest has reduced the below 8 lines of PHPUnit code to the above 1 line of code:

namespace Tests\Unit;
use PHPUnit\Framework\TestCase;

class ExampleTest extends TestCase
{
    /** @test */
    public function testBasicTest()
    {
        $this->assertTrue(true);        
    } 
    // ...
}

Improving the output

PHPUnit offers PHP users the ability to visualize the testing results in a very compact read:

While less verbosity is generally good, we felt we needed a bit more. Pest gave us more information per test:

For errors, Pest gives you direct access to the line of code that has failed:

Finally, you can get more info from the coverage option, if you want to display the number of lines of source code that have been tested and executed:

Adding scalability with datasets

As with most tests, the quality of the data is key. Along with using realistic and relevant mocked-up content, testing quality data requires:

Performing multiple tests on small and large variations of data
Ability to add new use cases without any recoding

With Pest’s “datasets”, you can create one test that takes an inline array of data. So, for example, you can test multiple emails, each one representing a different use case (maybe connectivity):

it('has emails', function ($email) {
   expect($email)->not->toBeEmpty();
})->with([
   'someone@jmail.com',
   'other@example.com'
]);

You can use multi-dimensional arrays as well:

it('has emails', function ($name, $email) {
   expect($email)->not->toBeEmpty();
})->with([
   ['Someone', 'someone@jmail.com'],
   ['Other', 'other@example.com']
]);

That’s how it’s done inline. You can also move the data outside of the test to gain more flexibility, For this, you’ll need the following folder structure:

—Tests
—---testemails.php
—Datasets
—---emails.php

Where the test functions go in folder Tests, and the datasets are defined in individual files in the Datasets folder. Here’s an example of the contents of the dataset file emails.php, which replaces the above inline email references:

dataset('emails', function () {
   Return (['other@example.com', 'enunomaduro@gmail.com'];
]);

And so the test function can now reuse emails.php for testing:

it('has emails', function ($email) {
   expect($email)->not->toBeEmpty();
   assertTrue(true);
})->with('emails');

And more

There are many additional features and options, as well as a number of assertions, expectations, and exceptions. And don’t forget that Pest is an extension of PHPUnit, so you can perform actions before and after a test, like setup in PHPUnit.

Here’s a last example. You can skip tests:

it('has home', function () {
   // ..
})->skip();

The skip function shows you how Pest really thinks about the tester and simplifies the important things. I’d advise you not to “skip” on Pest – it’s really a great addition to the PHP ecosystem.

Learn more about Pest, built by Nuno Maduro from @laravel.

DEV Community: Algolia

Setting up your MacBook for mobile Flutter development

Assumptions

Install Flutter

The iOS build path

Testing the iOS build path

The Android build path

Testing the Android build path

DevTool Intro: The Algolia CLI!

Meet the Algolia CLI!

Example #0: Set your active Algolia application with profiles

Example #1: Create an index and upload records from a file

Example #2: Managing index settings snapshots

Example #3: CI/CD pipelines that execute Algolia tasks

Get Involved!

Customize Algolia React components with Tailwind classes

Feature Spotlight: Trends Models in Recommend

Diving into Trends

Where to use Trends

Training your Trends model

Add trends to your frontend using the Recommend UI

Conclusion

Feature Spotlight: Optional Filters

Add autocomplete search to your Strapi blog

Prerequisites

Building the back end

Building the front end

Adding OAuth2 authentication to an AWS S3 static bucket with Okta

Discovering the solution with CloudFront and Lambda@Edge

Implementing the Lambda@Edge function

Lambda@Edge restrictions and caveats

1 – Environment variables

2 – Lambda package size limit

3 – Lambda region

4 – Lambda role permission

Authorization mechanism with Okta

Using JSON Web Tokens to store authorization result

Note on JWT expiration and renewal

Conclusion

Build a store locator that includes online sellers

The problem

Geographic data and Algolia Search

A single index solution

A multi-index solution

Preparing your indices

Building the front end

Querying for results

Putting it all together

Centralizing state and data handling with React Hooks: on the road to reusable components

How React Hooks saved our administration application

The requirements of an Admin application

A customized hook

How our useDatahook centralized the process

Calling the API

Caching the data

API keys vs JWT authorization – Which is best?

When API keys are fine

When it’s time to consider JWT authorization

Using API keys vs JWT authorization

Using Api Keys

Using the JWT token

A word about authentication & authorization

A word about better protection and security

What information is in a JWT token?

JWT authorization offers flexibility, reliability, and more security

Doing it with API keys

Switching over to JWT

In sum, sometimes JWT is absolutely needed and sometimes it’s overkill

JWT is overkill in the following two situations:

JWT is not overkill when …

Integrating custom dictionaries into NLP for greater clarity

Stop words and plurals, and compounds and segments – a few of my favorite things

Using dictionaries for text-based matching

I’m “down” with relevance

Ambiguity cloaked in “fur” (not squirrel fur)

The solution – Giving customers control with custom dictionaries

Normalizing our codebase and standardizing our dictionaries interface

Plugging in customer-created, custom dictionaries

Pest: a PHP testing framework that goes above and beyond PHPUnit

Reducing the testing codebase of PHPUnit with PHP chaining

How our `useData`hook centralized the process