DEV Community: Darasimi-Ajewole

Workflow Automation: Exploring AWS Step Functions Benefits

Darasimi-Ajewole — Thu, 25 Jul 2024 15:13:17 +0000

Business workflows inherently involve a sequence of interconnected tasks, often triggered by specific events or conditions. This natural alignment makes Event-Driven Architecture an ideal choice for automating and optimizing these workflows. However, the evolution of cloud computing has introduced a powerful solution that takes workflow automation to the next level — AWS Step Functions.

Benefits of AWS Step Functions

Simplicity and Visual Representation: AWS Step Functions provide a graphical console to design and visualize workflows. This visual representation simplifies the understanding of complex processes, making it easier for both developers and non-developers to collaborate on workflow design.
Built-In Error Handling: AWS Step Functions offer built-in error handling capabilities. This feature ensures that workflows gracefully handle failures and can be configured to retry or move to a specific error-handling step.
Stateful and Serverless Architecture: AWS Step Functions maintain the state of your workflow, eliminating the need for external storage or databases to track the progress of a process. Additionally, the serverless nature of Step Functions allows for automatic scaling, ensuring optimal performance without the need for manual intervention.
Easy Integration with AWS Services: Step Functions seamlessly integrate with a wide array of AWS services, such as AWS Lambda, AWS Batch, and AWS Fargate, enabling you to leverage the full capabilities of the AWS ecosystem in your workflows. This makes it easy to incorporate serverless compute, containerized applications, and other services into your automated processes.
Wait and Delay: Easily incorporate wait times and delays within your workflow without the need for elaborate queue management.
Cost-Efficiency: With AWS Step Functions, you pay primarily for the actual execution time of your workflows and a meagre fee for each state transition of your workflows. This pay-as-you-go model can result in cost savings compared to traditional EDAs, which may require more resources for handling asynchronous events.

Practical Examples

Order Processing Workflow

Let's explore using AWS Step function to automate an order processing workflow that consists of sequential steps for order validation, payment processing, and shipping confirmation.

    import * as cdk from 'aws-cdk-lib';
    import * as sfn from 'aws-cdk-lib/aws-stepfunctions';
    import * as tasks from 'aws-cdk-lib/aws-stepfunctions-tasks';
    import * as lambda from 'aws-cdk-lib/aws-lambda';
    import { Construct } from "constructs";

    export class OrderProcessingStack extends cdk.Stack {
      constructor(scope: Construct, id: string, props?: cdk.StackProps) {
        super(scope, id, props);

        // Define Lambda functions
        const validateOrderLambda = new lambda.Function(this, 'ValidateOrderLambda', { // Lambda Props});
        const processPaymentLambda = new lambda.Function(this, 'ProcessPaymentLambda', {});
        const confirmShippingLambda = new lambda.Function(this, 'ConfirmShippingLambda', {});

        // Define Step Functions workflow
        const definition = new sfn.StateMachine(this, 'OrderProcessingStateMachine');

        // Define tasks
        const validateOrderTask = new tasks.LambdaInvoke(this, 'ValidateOrder', { lambdaFunction: validateOrderLambda });
        const processPaymentTask = new tasks.LambdaInvoke(this, 'ProcessPayment', { lambdaFunction: processPaymentLambda });
        const confirmShippingTask = new tasks.LambdaInvoke(this, 'ConfirmShipping', { lambdaFunction: confirmShippingLambda });

        // Define final success state
        const successState = new sfn.Succeed(this, 'Order Processed Successfully');

        // Define sequence
        const orderProcessingDefinition = validateOrderTask
            .next(processPaymentTask)
            .next(confirmShippingTask)
            .next(successState);

        // Set the definition for the state machine
        definition.definition = orderProcessingDefinition;

        // Create Step Functions state machine
        new sfn.StateMachine(this, 'OrderProcessingStateMachine', {
          definition,
          timeout: cdk.Duration.minutes(5), // Set a timeout for the entire workflow
        });
      }
    }

    const app = new cdk.App();
    new OrderProcessingStack(app, 'OrderProcessingStack');

Enhanced Media Processing Workflow

Let's consider another example of a media processing workflow encompassing parallel tasks for video transcoding, thumbnail generation, and caption generation; as well as sequential steps for merging results and notification.

    import * as cdk from 'aws-cdk-lib';
    import * as sfn from 'aws-cdk-lib/aws-stepfunctions';
    import * as tasks from 'aws-cdk-lib/aws-stepfunctions-tasks';
    import * as lambda from 'aws-cdk-lib/aws-lambda';
    import { Construct } from "constructs";

    export class EnhancedMediaProcessingStack extends cdk.Stack {
      constructor(scope: Construct, id: string, props?: cdk.StackProps) {
        super(scope, id, props);

        // Define Lambda functions
        const transcodeVideoLambda = new lambda.Function(this, 'TranscodeVideoLambda', { // lambda props});
        const generateThumbnailLambda = new lambda.Function(this, 'GenerateThumbnailLambda', {});
        const generateCaptionLambda = new lambda.Function(this, 'GenerateCaptionLambda', {});
        const mergeResultsLambda = new lambda.Function(this, 'MergeResultsLambda', {});
        const notifyLambda = new lambda.Function(this, 'NotifyLambda', {});

        // Define Step Functions workflow
        const definition = new sfn.Parallel(this, 'MediaProcessingWorkflow')
          .branch(new tasks.LambdaInvoke(this, 'TranscodeVideo', { lambdaFunction: transcodeVideoLambda }))
          .branch(new tasks.LambdaInvoke(this, 'GenerateThumbnail', { lambdaFunction: generateThumbnailLambda }))
          .branch(new tasks.LambdaInvoke(this, 'GenerateCaption', { lambdaFunction: generateCaptionLambda }));

        // After the parallel tasks, merge results
        const mergeResultsTask = new tasks.LambdaInvoke(this, 'MergeResults', { lambdaFunction: mergeResultsLambda });

        // Notify after merging results
        const notifyTask = new tasks.LambdaInvoke(this, 'Notify', { lambdaFunction: notifyLambda });

        // Define final success state
        const successState = new sfn.Succeed(this, 'MediaProcessingCompleted');

        // Define workflow definition
definition.next(mergeResultsTask).next(notifyTask).next(successState);

        // Create Step Functions state machine
        new sfn.StateMachine(this, 'EnhancedMediaProcessingStateMachine', {
          definition,
          timeout: cdk.Duration.minutes(20), // Set a timeout for the entire workflow
        });
      }
    }

    const app = new cdk.App();
    new EnhancedMediaProcessingStack(app, 'EnhancedMediaProcessingStack');

These examples showcase the flexibility and power of AWS Step Functions in simplifying the orchestration of complex workflows efficiently.

In conclusion, By harnessing AWS Step Functions, businesses can streamline their processes, enhance collaboration, and achieve cost-effective and scalable automation. To learn more about AWS Step Functions, you can get started with the official docs.

While it can be a game-changer for complex workflows, it's essential to note that it might be overkill for simpler scenarios with minimal orchestration needs. As with any tool, understanding the specific requirements of your business processes is key to choosing the right solution.

Scaling Up: Hot-Reloading Multiple TypeScript Lambdas in LocalStack

Darasimi-Ajewole — Fri, 26 Jan 2024 12:51:10 +0000

In a recent consultancy project, I was tasked with creating a mock environment for an asynchronous webservice using LocalStack, with the primary aim of minimizing the feedback loop associated with integrating our systems. This involved using AWS CDK to provision multiple TypeScript runtime lambdas in LocalStack and esbuild to build the functions.

On completing the mock environment, I decided to go the extra mile by attempting to enable hot reloading for the lambdas, a feature that proved more challenging than expected, as outlined in the official LocalStack guide.

After navigating through moments of frustration, fueled by a few cups of Yorkshire tea, I eventually uncovered the solution to hot-reloading multiple TypeScript lambdas in LocalStack. Below, I have outlined guide on how to achieve this for others navigating similar challenges.

Before diving into this guide, it's recommended to familiarize yourself with the official hot-reloading Localstack guide for a single Lambda function.

The following steps provide a summary of the official guide for hot-reloading TypeScript Lambda functions in LocalStack:

The Build Step:
package.json

"scripts": {
        "build": "esbuild index.ts \
                   --bundle --minify --sourcemap \
                   --platform=node --target=es2020 \
                   --outfile=dist/index.js --watch"
    },

Executing the above build script with npm run build uses esbuild to bundle and minify the TypeScript code into a single JavaScript file, stored in the dist folder.

Deploying to LocalStack

awslocal lambda create-function \
   --function-name hello-world \
   --runtime "nodejs16.x" \
   --role arn:aws:iam::123456789012:role/lambda-ex \
   --code S3Bucket="hot-reload",S3Key="$(PWD)/dist" \
   --handler index.handler

If followed correctly, these steps should enable hot-reloading for a single Lambda function after every code change.

Hot-Reloading Multiple Lambdas

To extend hot-reloading to multiple Lambdas, a key modification is needed at the build step:

"scripts": {
        "build": "esbuild lambdas/src/*.ts 
                    --bundle --minify --sourcemap \
                    --platform=node --target=es2020 \
                    --outdir=lambdas/build --watch"
    },

This command individually transpiles, bundles, and minifies each Lambda entry file in the lambdas/src folder, placing the output in the lambdas/build directory.

Note: The lambdas/src folder should match the directory where your Lambdas reside. Other bundlers, not just esbuild, can be used for the build step.

Proceed to deploy each Lambda to Localstack using the awslocal lambda create-function command, and they should hot-reload with every code change.

Important: When deploying from within a container, ensure that the S3key is a path on the host, as Localstack maps directories from the host to the Lambda containers.

Bonus Information: AWS Cloud Development Kit (CDK) Configuration for Hot-Reloaded Lambda.

import { NodejsFunction, NodejsFunctionProps } from 'aws-cdk-lib/aws-lambda-nodejs';
import { Architecture, Code, Function, Runtime } from "aws-cdk-lib/aws-lambda";
import { Bucket } from "aws-cdk-lib/aws-s3";
import { Construct } from "constructs";
import { join, basename } from "path";


const STAGE = process.env.STAGE ?? "local";
const LAMBDA_MOUNT_CWD = join(__dirname, "lambdas/build")

function ApplicationFunction(
  scope: Construct,
  id: string,
  props: NodejsFunctionProps
) {
  if (STAGE === "local") {
    return LocalFunction(scope, id, props);
  }
  return new NodejsFunction(scope, id, props);
}

function LocalFunction(
  scope: Construct,
  id: string,
  props: NodejsFunctionProps
) {
  const hotReloadBucket = Bucket.fromBucketName(
    scope,
    `HotReloadingBucket-${id}`,
    "hot-reload"
  );

  if (!props.entry) throw new Error('Entry point is required');

  const fileName = basename(props.entry, ".ts");
  const handler = props.handler ?? "handler";
  const runtime = props.runtime || Runtime.NODEJS_18_X;

  return new Function(scope, id, {
    ...props,
    code: Code.fromBucket(hotReloadBucket, join(__dirname, "lambdas/build")),
    runtime,
    handler: `${fileName}.${handler}`,
  });
}

Bootstrap and deploy to LocalStack by running:

cdklocal bootstrap && cdklocal deploy --require-approval never

For a complete working example of hot-reloading multiple TypeScript Lambdas, refer to this example CRUD application with five hot-reloaded Lambdas.

May your Lambdas always hot-reload!

Page Model Design Pattern: Writing maintainable End-to-End Test

Darasimi-Ajewole — Fri, 19 Jan 2024 13:05:41 +0000

End-to-End (E2E) testing is a crucial part of the software development lifecycle, ensuring that all components of an application work seamlessly together. However, as applications grow in complexity, maintaining E2E tests becomes challenging. The Page Model Design Pattern is an excellent solution to address this challenge, providing a structured and maintainable approach to E2E test architecture.

Understanding the Page Model Design Pattern

The Page Model Design Pattern is an architectural pattern that encourages the creation of a model for each web page or component in the application. This model encapsulates the interactions and elements on that specific page, making it easier to manage and maintain E2E tests.

How to Create End-to-End Tests Using the Page Model Design Pattern

Step 1: Page Model Creation
Begin by creating a page model for each significant page or component in your application. This model should include methods to interact with the elements on that page. For example:

 // Example Page Model for a Login Page (loginPage.ts)
import { Selector, t } from 'testcafe';
import DashboardPage from './dashboardPage';

export default class LoginPage {
    private usernameInput: Selector = Selector('input').withText('username');
    private passwordInput: Selector =  Selector('input').withText('password');
    private loginButton: Selector = Selector('button').withText('login');

    async enterCredentials(username: string, password: string): Promise<void> {
        await t.typeText(this.usernameInput, username)
               .typeText(this.passwordInput, password);
    }

    async clickLoginButton(): Promise<void> {
        await t.click(this.loginButton);
    }

    async login(username: string, password: string): Promise<DashboardPage> {
        await this.enterCredentials(username, password);
        await this.clickLoginButton();
        return new DashboardPage();
    }
}

// Example Page Model for a Dashboard Page (dashboardPage.ts)
import { Selector } from 'testcafe';

export default class DashboardPage {
    private welcomeMessage: Selector = Selector('[data-id="welcome-message"]');

    async getWelcomeMessage(): Promise<string> {
        return await this.welcomeMessage.innerText;
    }
}

Step 2: Test Implementation

With the page models in place, writing tests becomes more straightforward and focused. Tests can now leverage the methods provided by the page models. For example:

 // Example E2E Test Using Page Models (e2eTest.ts)
import { Selector } from 'testcafe';
import LoginPage from './page-models/loginPage';
import DashboardPage from './page-models/dashboardPage';

fixture`Login Feature`.page`https://your-app-url.com`;

test('Successful Login and Navigate to Dashboard', async (t) => {
    const loginPage = new LoginPage();

    // Log in and get the DashboardPage instance
    const dashboardPage: DashboardPage = await loginPage.login('your_username', 'your_password');

    // Assertion for a successful login
    await t.expect(dashboardPage.getWelcomeMessage()).contains('Welcome');

    // Add additional actions on the DashboardPage if needed
    // For example: await dashboardPage.someAction();
});

Benefits of Page Model Design Pattern

Modularity: Each page or component has its own model, promoting modularity and making it easier to update tests when the UI changes.
Readability: Tests become more readable and expressive, as the interaction with elements is encapsulated within page models.
Reusability: Page models can be reused across multiple tests, reducing duplication and promoting a DRY (Don't Repeat Yourself) approach.
Maintainability: When the UI changes, updates are confined to the corresponding page model, minimizing the impact on the entire test suite.

As your applications expand, this pattern ensures your E2E tests remain strong and adaptable, consequently improving the overall quality of your software. Happy Testing!

The Crucial Role of Load Balancers for App Engine Applications

Darasimi-Ajewole — Fri, 19 Jan 2024 12:51:42 +0000

Google App Engine (GAE) is a powerful platform for developing and deploying scalable applications, but even with its robust features, there are instances where integrating a load balancer becomes essential for optimizing performance and managing complexities. This article delves into the reasons why App Engine applications still benefit from the inclusion of a load balancer:

Obtaining a Unique IP Address
A pivotal reason App Engine applications often necessitate a load balancer lies in the inherent shared IP address structure of GAE apps. Shared IPs bring challenges like IP Reputation issues and limited control. Load balancers step in to address these challenges head-on by providing a unique IP address for GAE apps. This not only enhances control and management but also fortifies the security of the application.
Efficient Dispatching Across Multiple Services
While App Engine enables the deployment of multiple services within an application, the native dispatch.yaml file caps at 20 routes, often insufficient for larger applications. In contrast, load balancers excel, handling a minimum of 1,000 routes and up to 1,000,000 routes with proper configuration. This scalability provides an ideal solution for applications with diverse service needs.
Simplified Circuit Breaker Configuration
Load balancers simplify the configuration of circuit breakers, enhancing the overall reliability of the application. Configuring circuit breakers directly on the load balancer is often more straightforward than implementing them within the application code. This ease of configuration ensures better fault tolerance and responsiveness, crucial for maintaining a stable user experience.
Streamlined Management for Multiplatform Applications
For applications hosted on various platforms such as Cloud Functions, Cloud Run, and Cloud Storage, a load balancer acts as a centralized control point. Unlike having to add custom rules on your domain provider list for each service, load balancers provide a unified interface for managing traffic, making it more efficient to handle diverse services seamlessly.
Finer Traffic Control based on Different Parameters
App Engine's native tools, like the dispatch.yaml file, have limitations in achieving fine-grained traffic control, particularly concerning parameters such as headers, subdomains, and URL query strings. Notably, the GAE front end server exclusively directs domain traffic to the default version, restricting more detailed traffic management. Load balancers step in as a solution by offering precise traffic control for App Engine applications. They enable granular traffic control that can be based on headers, subdomains, and URL query strings, even allowing the forwarding of domain traffic to specific versions. This empowerment facilitates robust A-B testing and empowers developers to optimize application performance with meticulous parameter-based control.

Convinced and would like to get started? Follow the steps below to integrate a load balancer with your App Engine application.

Quick Start with Load Balancers for App Engine:

Navigate to Google Cloud Console: Begin by logging into the Google Cloud Console.
Select Your Project: Ensure you have an active project or create a new one.
Access Load Balancers: In the left navigation pane, go to "Networking" and select "Load balancing."
Create a New Load Balancer: Click on the "Create" button to set up a new load balancer.
Configure Load Balancer: Follow the prompts to configure your load balancer. Specify the backend services, health checks, and other relevant settings.
Target App Engine Services: Connect your load balancer to the App Engine services you want to manage. Ensure proper routing and distribution of traffic.
Review and Deploy: Review your configuration settings and deploy the load balancer. Deployed changes could take as much as a minute before they reflect.
Monitor and Optimize: Once deployed, regularly monitor your load balancer's performance through the Google Cloud Console. Adjust settings as needed to optimize based on application requirements.

For detailed instructions and additional information, you can also refer to the Google Cloud documentation on Load Balancing with App Engine.

Modelling with NDB

Darasimi-Ajewole — Tue, 08 Jun 2021 18:43:41 +0000

Python ORMs(Object Relationship Mappers) for SQL databases, such as Django ORM and SQLAlchemy, ship with properties that explicitly model database relationships between different database tables.
However, ndb - the python ORM built for Google Cloud Datastore does not provide such properties. Fortunately, using existing ndb properties, database relationships can still be modelled with ndb.
There are three types of relationships in a database, namely: one-to-one, one-to-many, and many-to-many. Let's dive into each relationship's implementation using ndb as well as their pitfalls.

Assumptions:
1) Readers are familiar with Database Relationships.
2) Readers have basic familiarity with python.
3) Readers are familiar with various column properties of ndb.

One-to-One

Famous examples of one-to-one relationships include:

Western Countries and their presidents.
Personal bank account and its owners.
Monogamous marriage between two partners.

To implement a one-to-one relationship, ndb provides the KeyProperty for storing datastore keys. The KeyProperty also accepts an optional Kind argument to validate keys assigned to this property.
One-To-One relationships between two ndb Kinds can be implemented by creating a KeyProperty on one of the Kinds and setting its optional kind argument to be the other Kind.
Let's write some code to illustrate how KeyProperty can be used to implement a one-to-one relationship:

Class Person(ndb.Model):
    name = ndb.StringProperty()
    nationality = ndb.StringProperty(repeated=true)
    email = ndb.StringProperty()
    age = ndb.IntegerProperty()

Class Country(ndb.Model):
    name = ndb.StringProperty()
    location = ndb.GeoProperty()
    citizen_count = ndb.IntegerProperty()
    currency_name = ndb.StringProperty()
    president = ndb.KeyProperty(Person)

# Creating
obama = Person(name='Barack Obama', 
               nationality=['American',  'Kenya'], 
               email='darasimiajewole@gmail.com', 
               age=61)
obama_key = obama.put()

america = Country(name='America', 
                  citizen_count=1*10**6, 
                  currency_name='dollars', 
                  president=obama_key)

america_key = america.put()
america_id = america_key.id()

# Reading
america = Country.query(Country.president=obama_key).get()
obama = america.president.get()

Pitfalls

KeyProperty does not enforce uniqueness like the one-to-one fields of traditional SQL ORMs. Unless a uniqueness check is being made before the entities of the Two Kinds are created/updated, a one-to-many relationship can be inadvertently created instead of one-to-one.

One-to-Many

Famous examples of one-to-many relationships include:

Polygamous marriages.
Father and his Children.
Planets and their Moons
Companies and their products
Countries and their States

You have two ways to model a one-to-many relationship with ndb:

1) Using KeyProperty

This is quite similar to modeling a one-to-one relationship except the uniqueness check is omitted.
To implement this relationship between two Kinds using KeyProperty, simply create a field on one of the Kinds with the property type as the KeyProperty type and pass the other Kind class as the optional type argument. Let's dive into some code to illustrate the modeling:

Class Country(ndb.Model):
    name = ndb.StringProperty()
    location = ndb.GeoProperty()
    citizen_count = ndb.IntegerProperty()
    currency_name = ndb.StringProperty()

Class State(ndb.Model):
    name = ndb.StringProperty
    location = ndb.GeoProperty()
    population = ndb.IntegerProperty()
    country = ndb.KeyProperty(Country)

# Creating
## parent entity
nigeria = Country(name='Nigeria',       
                  citizen_count=2*10**7,   
                  currency_name='Naira')
nigeria_key = country.put()

## child entities
lagos = State(name='Lagos', 
              population=30*10**6, 
              country=nigeria_key)
lagos.put()

abj = State(name='Abuja',  
            population=10*10**6, 
            country=nigeria_key)
abj.put()

# Reading all states in a country
all_state = State.query(State.country == nigeria_key).fetch()
#[lagos, abj]


# Updating
china = Country(name='China',        
                citizen_count=2*10**9,   
                currency_name='yen')
china_key = china.put()

lagos.country = china_key
lagos.put()

Pitfalls

Querying for the child entities is not supported within a datastore transaction because they don't have the same "datastore ancestor".

2) Modelling using Ancestor Key

Just a bit of background on datastore and ancestor keys culled from the official docs:
Datastore entities are identified by a key, an identifier unique within the application's datastore. The key can have an ancestor, which is another key. This parent can have an ancestor, and so on; at the top of this chain of parents is a key with no ancestor, called the root. Entities whose keys have the same root form an entity group or group.
Leveraging on the ancestor key relationship between root entities and entity groups, one-to-many relationships can be established between two Kinds by making the Key of the parent Kind the ancestor to the Key of the child Kind. Let's show some code to illustrate this:


Class Company(ndb.Model):
    name = ndb.StringProperty()
    staff_num = ndb.StringProperty()
    headquarter = ndb.StringProperty()

Class Product(ndb.Model):
    name = ndb.StringProperty()
    market_share = ndb.IntegerProperty()
    users_num = ndb.StringProperty()


# Creating
## parent
google = Company(name='Google', 
                 staff_num=20000, 
                 headquarter='Yabacon Valley')
## parent key
google_key = google.put()

## child key will have parent as google_key
gmail_key = ndb.Key(Product, 'Gmail Service', parent=google_key)
chrome_key = ndb.Key(Product, 'Chrome Browser', parent=google_key)
## child
gmail = Product(key=gmail_key, 
                name='Gmail', 
                market_share=80, 
                users_num=200)
gmail.put()
chrome = Product(key=chrome_key, 
                 name='Chrome Browser',
                 market_share=70,
                 users_num=34)
chrome.put()

# Reading
## reading parent entity
google_key = chrome.key.parent()

## reading child entities
all_product = Product.query(ancestor=google_key).fetch()
print(all_product) # [chrome, gmail]

To change the parent entity in a one-to-many relationship modeled using an Ancestor key, the child entity is deleted and recreated with a new key formed from the new parent key. These extra steps are needed because entity keys are immutable in the datastore after been created. Let's show some code:

# updating
## new parent entity
microsoft = Company(name='Microsoft', 
                    staff_num=5000, 
                    headquarter='Palo Alto')
microsoft_key = microsoft.put()

# child entity
chrome = chrome_key.get()
chrome_key.delete()

## creating new child entity key from new parent key
new_chrome_key = ndb.Key(Product,
                         'Chrome Explorer', 
                          parent=microsoft_key)
new_chrome = Product(key=new_chrome_key)
new_chrome.name = chrome.name
new_chrome.market_share = chrome.market_share
new_chrome.users_num = chrome.users_num
new_chrome.put()

Pitfalls:

Datastore writes take longer, due to replication of changes across the entity groups stored on many Datastore servers. This could lead to inconsistent reads seconds after a datastore write.
It's highly advisable not to use the keys of the child entities as parent keys of entities of another Kind. This is due to the volatility of the child entity keys during the update of its parent.
The relationship is not explicit enough from the model definition.
Multiple One-to-Many relationships cannot be formed by a child entity to multiple parents of a different kind. So the relationship between a User and their home country, and the same User and their Job, cannot be modeled using ndb ancestor keys.

Many-to-Many relationships

Famous examples of this relationship include:
1) Pizza and their toppings. Pizzas can have multiple toppings and a topping can be on multiple pizzas.
2) Citizens with multiple countries.

The KeyProperty type from NDB has earlier been used to implement the one-to-one relationship and also one-to-many relationship, it should not be surprising that it can also be used to model the many-to-many relationship.
That is possible because the KeyProperty type supports the repeated argument, which means a list of Datastore keys can be attached to the KeyProperty fields.
Leveraging on that, many-to-many relationships between two NDB kinds such as Pizza and Toppings could be modeled using a repeated KeyProperty on the Pizza kind with the Kind argument as the Toppings kind. Below is some code to illustrate how the modeling works:

class Pizza(ndb.Model):
    name = ndb.StringProperty()
    steps = ndb.TextProperty(repeated=true)
    toppings = ndb.KeyProperty(Toppings, repeated=true)

Class Toppings(ndb.Model):
    name = ndb.StringProperty()
    ingredients= ndb.StringProperty(repeated=true)
    date_created = ndb.DatetimeProperty(auto_now=true)

# Creating
cheese = Toppings(name='cheese', 
                  ingredients= ['cheese', 'random seasoning'])
onions = Toppings(name='onion', 
                  ingredients= ['onions', 'chili seasonings'])
bacon = Toppings(name='bacon', 
                 ingredients= ['bacon', 'garlic seasoning'])
ndb.put_multi(cheese, onions, bacon)

macaroni_cheese = Pizza(name='Macaroni Cheese', 
                        steps=['A', 'B', 'C'], 
                        toppings=[cheese.key, bacon.key])
barbeque = Pizza(name='Barbeque', 
                 steps=['H', 'L', 'O'], 
                 toppings=[cheese.key, onion.key])
ndb.put_multi(macaroni_cheese, barbeque)

# Reading
barbeque = Pizza.get_by_id(barbeque_id)
toppings = barbeque.toppings
### [cheese.key, onion.key]

cheese_pizzas = Pizza.query(Pizza.toppings == cheese.key).fetch()
### [macaroni_cheese_key, barbeque_key]


# Updating
sweet_corn = Toppings(name='sweet corn', 
                      ingredients=['sweet corn', 'honey'])
sweet_corn.put()


macaroni_cheese.toppings.append(sweet_corn.key)
macaroni_cheese.put()

Pitfalls

Having more than one indexed repeated properties on a Kind can lead to exploding index. Due to this behavior, it's advisable for ndb kinds not to have more than a single Many-to-Many relationships.

Real-time update strategy in web

Darasimi-Ajewole — Tue, 08 Jun 2021 18:43:08 +0000

In the modern world of sleek interfaces and highly interactive web applications, giving a real-time update of the latest information to users is fast becoming a necessity for a fascinating user experience. While real-time updates are a plus for blogs and news feeds, it's an absolute necessity for time-sensitive web apps such as chat apps.
Real-time update strategies fall into two categories: Polling and Push. Polling strategies involve querying the server for new updates at intervals. Push strategies mean pushing new updates to the client by the server.
Here is an overview of various strategies for implementing real-time updates in web applications, starting with those under the polling category:

1) Window Refreshes at intervals:

This strategy involves reloading the entire web app at an interval to fetch the latest changes on the server-side. This strategy works fine for web apps that are server-side rendered. To accomplish this, you can either add the HTML meta tag for window refresh or use the window.reload javascript API. Code samples to refresh a web app every 30 seconds are present below:

<head>
  <meta http-equiv="refresh" content="30">
</head>

setInterval(function() { location.reload() }, 30000 )

2) AJAX:

AJAX was a breakthrough improvement to the web that allowed web apps to make requests to servers without reloading the whole page.
Using AJAX for real-time updates involves making requests to servers for new updates at intervals and updating the web app with new updates. It's a simple enough approach that would be sufficient for most features that depend on real-time updates. The recommended strategy for web apps where a delayed real-time update will not significantly impair user experience or update interval is predictable.
Check out the gist below on how to implement a real-time update of 2 seconds with AJAX.

3) Websocket:

For web apps such as the stock markets reporting app and chats, where updates can be erratic and delayed updates can lead to a poor user experience, the underlying technical shortcomings with AJAX real-time updates begin to surface. In an attempt to fix these issues, WebSocket got introduced to the web.
With WebSocket, web apps can maintain a persistent connection with the server. WebSocket connections stay alive after receiving responses, unlike the short-lived request-response connection used by AJAX. Servers can push new updates to web apps via web sockets without the constraints of the traditional request-response cycle.
Check out this article on how to implement a WebSocket connection between a webpage and a NodeJS server

4) Managed Real-time Update Service

While WebSocket fixed the shortcomings with AJAX, it became an expensive approach regarding implementation and server-side cost. In response to the issues with AJAX and WebSockets, real-time update services emerged to manage the cost of managing real-time updates on the web and simplify the implementation of real-time updates so developers can focus on the logic of their web apps.
Real-time update service acts as a proxy between web app servers and the web app client. They receive new updates from the servers and oversee the push to the client. They manage real-time update concerns ranging from scaling of real-time connection to timeouts, dropoffs, persistent connection reconnect, fall back to AJAX from WebSockets, offline synching, among many others. An inexhaustive list of real-time update service include:

Pusher
Firebase cloud messaging
Firebase Realtime Database
Firebase Cloud Firestore
PubNub
OneSignal
Twilio

P.S: For production usage, check out the pricing of these services

5) WebRTC

A relatively new introduction to the web touted as a peer-to-peer technology that enables direct real-time communications in web apps. It was introduced to the web to address technical constraints faced by web apps that don't fall into the client-server model of HTTP-Web - examples of such web apps include chat apps, music sharing web app, multicast streaming services.
WebRTC eliminates the need to have an intermediate server that maintains the connection between different users of a peer-to-peer web app, hence shaving off the extraneous latency introduced by servers in peer to peer web applications. For an intro to WebRTC, check this article

I hope this article has given a good overview of various strategies for implementing real-time updates in web apps. Kindly comment on what you think.

Integrating Jest with Gulp

Darasimi-Ajewole — Thu, 03 Jun 2021 19:00:35 +0000

The Gulp-jest plugin for running jest tests are extremely limited in their integrations with gulp workflow. Fortunately, Jest provides jest-cli, a library that allows you to run jest tests programmatically and also returns the result of the test. Taking advantage of this package, you can customize the operation of your gulp workflow.

To get started, install the jest-cli in your project:

npm i jest-cli
yarn add jest-cli

To run jest tests programmatically with jest-cli, a sample code on how this is done is provided below:

const jest = require('jest-cli');
const testRunner = async () => {
  const { results } =  await jest.runCLI(
    {json: false}, // jest cli options, check out more on https://jestjs.io/docs/cli
    ['project_directory'] // array of project directories in order to run multiple project tests
  )
  console.log(results);
  return results;
}

testRunner();
// clipped result
`{
    numFailedTestSuites: 0,
    numFailedTests: 0,
    numPassedTestSuites: 5,
    numTotalTestSuites: 7,
    numTotalTests: 98,
    success: true,
    testResults: [
      [Object]
    ],
    wasInterrupted: false
 }`

Please note, the code above should work on versions of jest-cli <= 24.9.0, but might fail to work on newer versions of jest-cli, this is due to runCLI which is a private method not guaranteed to have the same behaviour across all versions of jest-cli.

From the returned results of the jest.runCLI, you can write a lot more jest integrations with your gulp workflow, and customise the behaviour of the workflows better than gulp-jest allows.
A popular integration of test frameworks with gulp workflows would be aborting workflows if tests don't pass, a simple way to implement this with the help of jest-cli is shown below:

// gulpfile.babel.js
const jest = require('jest-cli');

async function testJS() {  
   const testResults =  await jest.runCLI({json: false},['project_directory'])
   const { results } = testResults
   const isTestFailed = !results.success;
   if (isTestFailed) {
       console.log('You have some failed test cases, kindly fix')
       process.exit() // Breaks Gulp Pipe
   }
}

gulp.task('build', gulp.series(
  testJS,
  buildAssets,
  // Any other gulp tasks
))

With the implementation above, Test checks of Jest test is fully integrated into the gulp build workflow.

Too long did not read:
1) Install jest-cli in your project.

yarn add jest-cli

2) Use jest-cli to run your Jest tests programmatically. A sample code is shown below plus the returned result.

const jest = require('jest-cli');
jest.runCLI({json: false},['project directory'])
    .then((testResults) => console.log(testResults))

3) Use the result from the jest-cli to customise your gulp workflow. check a sample of how it's done here