DEV Community: Functional Javascript

MongoDB - Slicing & Dicing - Get Daily Activity - Series #15

Functional Javascript — Fri, 19 Feb 2021 04:59:50 +0000

Intro

We can easily perform data analysis on our data by slicing our data into buckets based on dates.

This query retrieves a report on how many "likes" per day were performed by users over the last 30 days.

  mgArr(dbEnum.nlpdb, collEnum.users_actions,
    matchExact("actionCateg", "fav"),
    matchExact("isFav", true),
    {
      $group: {
        _id: {
          year: { $year: "$_id" },
          month: { $month: "$_id" },
          day: { $dayOfMonth: "$_id" }
        },
        count: { $sum: 1 },
   }
    },
   lastInserted(30),
   )

The Output:

/*
[
  { _id: { year: 2021, month: 2, day: 18 }, count: 123 },
  { _id: { year: 2021, month: 2, day: 17 }, count: 2214 },
  { _id: { year: 2021, month: 2, day: 15 }, count: 369 },
  { _id: { year: 2021, month: 2, day: 14 }, count: 1353 },
  { _id: { year: 2021, month: 2, day: 12 }, count: 492 },
  { _id: { year: 2021, month: 2, day: 11 }, count: 1230 },
  { _id: { year: 2021, month: 2, day: 10 }, count: 1476 },
  { _id: { year: 2021, month: 2, day: 9 }, count: 4305 },
  { _id: { year: 2021, month: 2, day: 8 }, count: 1722 },
  { _id: { year: 2021, month: 2, day: 6 }, count: 984 },
  { _id: { year: 2021, month: 2, day: 5 }, count: 2706 },
  { _id: { year: 2021, month: 2, day: 4 }, count: 861 },
  { _id: { year: 2021, month: 2, day: 3 }, count: 369 },
  { _id: { year: 2021, month: 2, day: 2 }, count: 2829 },
  { _id: { year: 2021, month: 2, day: 1 }, count: 1230 },
  { _id: { year: 2021, month: 1, day: 29 }, count: 615 },
  { _id: { year: 2021, month: 1, day: 28 }, count: 3444 },
  { _id: { year: 2021, month: 1, day: 27 }, count: 369 },
  { _id: { year: 2021, month: 1, day: 26 }, count: 2706 },
  { _id: { year: 2021, month: 1, day: 25 }, count: 123 },
  { _id: { year: 2021, month: 1, day: 24 }, count: 1599 },
  { _id: { year: 2021, month: 1, day: 23 }, count: 246 },
  { _id: { year: 2021, month: 1, day: 22 }, count: 3567 },
  { _id: { year: 2021, month: 1, day: 21 }, count: 1353 },
  { _id: { year: 2021, month: 1, day: 20 }, count: 1722 },
  { _id: { year: 2021, month: 1, day: 19 }, count: 3813 },
  { _id: { year: 2021, month: 1, day: 18 }, count: 2091 },
  { _id: { year: 2021, month: 1, day: 17 }, count: 3321 },
  { _id: { year: 2021, month: 1, day: 16 }, count: 123 },
  { _id: { year: 2021, month: 1, day: 14 }, count: 246 }
]
*/

Notes

The query is using the MongoDB Aggregation Framework.

The query pipeline is just an array of query stages.

The stages are query syntax that is in a JavaScript object format.

So each stage is a JS object that describes that portion of the query that's being composed.

Thus we compose a query together by adding stages.

As an analogy to SQL syntax, you can think of stages as SQL clauses like the SELECT clause, the WHERE clause, or the GROUP BY clause.

The matchExact stage is my wrapper func, which returns the raw stage object syntax.
The raw syntax would look like....

{ $match: { actionCateg: "fav", isFav: true  } }

The $group stage has an _id key. This DOES NOT represent the Primary Key (PK) of the Collection. It represents the uniquely grouped item in the grouping. In this case it's the combination of "year, month and day" that represents the unique record being summed.

The "$_id" value actually represents the PK of the Collection. As you remember from the last article, the PK contains the timestamp information. The $year operator (think of these Mongodb operators as built-in utility functions) extracts the year part of the timestamp which is stored in the "$_id" (PK).

In the last stage (lastInserted, my utility func) we retrieve the last 30 days.

What's Next

More magic with MongoDB coming up in future series articles.

As always, if you have an questions or input, let me know.

MongoDB Marvels - Slicing Data by Dates - Series #14

Functional Javascript — Wed, 17 Feb 2021 23:24:13 +0000

Intro

Let's look at a simple query first as we prep ourselves for more thorough queries in this Series:

Query for how many user actions per hour block.
This will report which hour blocks the site is busiest.

  mgArr(dbEnum.nlpdb, collEnum.users_actions,
    {
      $addFields: {
        hourUtc: {
          $hour: { date: "$_id" }
        },
        hourPst: {
          $hour: { date: "$_id", timezone: "-08:00" },
        },
      },
    },
    groupByKey("hourPst"),
    sortDesc("count"),
  )

Notes

The server timestamp is in UTC Time (Coordinated Universal Time).

The db query code also converts the timezone from UTC to PST.
(In another post we'll extract the user's timezone from their device.)

The query has 3 stages:

The $addFields stage adds two computed fields to our resultset, "hourUtc" and "hourPst". (Just to show an example of how to get dates without and with an offset)

The Group stage will pivot on the hour component of the date, for the PST timezone.

Sort Descending, so we see the busiest hours at the top.

We will get back 24 records (24 hours in each day), so we don't need to Paginate.

The output:

/*
 { count: 610, hourPst: 15 },
  { count: 58, hourPst: 20 },
  { count: 55, hourPst: 21 },
  { count: 51, hourPst: 23 },
  { count: 49, hourPst: 14 },
  { count: 41, hourPst: 10 },
  { count: 38, hourPst: 13 },
  { count: 34, hourPst: 22 },
  { count: 33, hourPst: 18 },
  { count: 32, hourPst: 16 },
  { count: 27, hourPst: 17 },
  { count: 24, hourPst: 11 },
  { count: 23, hourPst: 0 },
  { count: 17, hourPst: 19 },
  { count: 17, hourPst: 1 },
  { count: 15, hourPst: 12 },
  { count: 9, hourPst: 3 },
  { count: 5, hourPst: 2 },
  { count: 5, hourPst: 8 },
  { count: 3, hourPst: 6 }
*/

Notes

Looks like 3:00:00 to 3:59:59 PM is very busy for some reason. :)

Though this is an example query, you can see how you can get meaningful and sometimes surprising insights into your data by carving it up into timeseries dimensions.

What's Next

We'll cover more power with dates and time series in this Series of Articles.

As always, if you have an questions or input, let me know.

MongoDB Marvels - Data Between Dates - Series #13

Functional Javascript — Tue, 16 Feb 2021 18:43:32 +0000

Intro

Let's look at a MongoDB query that uses the Mongo Aggregation Framework (AF) that retrieves a resultset of docs between two points in time.

mgArr(dbEnum.nlpdb, collEnum.users_actions,
    { $addFields: { ts: { $toDate: "$_id" } } },
    {
      $match: {
        ts: {
          $gte: new Date("2021-02-01T00:00:00.000Z"),
          $lt: new Date("2021-02-15T00:00:00.000Z")
        }
      }
    },
    { $sort: { ts: 1 } },
    { $skip: 0 },
    { $limit: 100 },
  )

Notes

There are 5 stages here.

extract the timestamp from the _id field (PK), as a new field called "ts" (timestamp)
filter (match) by dates that are greater than the earliest date and less than the latest date
sort by the earliest date (the 1 means ascending; a -1 would mean descending)
let's add pagination, which requires two stages, $skip and $limit. $skip: 0 mean get the first page of results without skiping to the next page
limit the resultset to the first 100 results, as page 1 (no skip)

These 5 stages are the raw AF (Aggregation Framework) syntaxes.

Version 2 - using functional wrappers

  mgArr(dbEnum.nlpdb, collEnum.users_actions,
    addField_ts(),
    matchBetweenDates("2021-02-01", "2021-02-04"),
    sortAsc("ts"),
    paginate(100, 1),
  )

Notes

We've made the code more terse and increased its readability by abstracting away the tedious and repetitious curl braces. Now it looks more functional, since it's just one function call after another, passing in our argument that are particular to this queries use case.

Notice the naming convention for the addField_ts func. The underscore indicates a namespacing. I use namespacing to clearly specify a category of funcs. I have a number of "addField" funcs that add fields to a resultset for particular common use cases. Adding a new timestamp field is just one common use case.

Notice it appears we have 4 stages now instead of 5. The paginate func composes two stages, $skip and $limit to give us a simpler pagination behavior. The paginate func returns the $skip and $limit, so technically there are still 5 stages sent to MongoDB as a raw query.

I made the matchBetweenDates func to be inclusive of the start and end date.
Here is the source code:

/**
@func
get docs between two dates
inclusive of days

@param {string} start - the earliest day to include
@param {string} end - the latest day to include
@return {object}
*/
export const matchBetweenDates = (start, end) => {
  return {
    $match: {
      ts: {
        $gte: new Date(start + "T00:00:00.000Z"),
        $lte: new Date(end + "T23:59:59.000Z")
      }
    }
  };
};

What's Next

We'll be covering more advanced usages of partitioning your data on date dimensions, to allow you better insights into your data, and make better predictions about the future.

As always, if you have any questions on this let me know.

MongoDB Marvels - Dancing with Dates - Series #12

Functional Javascript — Mon, 15 Feb 2021 19:47:46 +0000

MongoDB Marvels - Dancing with Dates - Series #12

MongoDB - Working with dates

Intro

If you insert a doc with no explicit PK (primary key) — which is by default a field with a _id key — Mongo will insert an autogenerated value for you.

One useful feature is that embedded in that PK is the ISODate.

We can extract it like this:

  mgArr(dbEnum.nlpdb, collEnum.users_actions,
    { $addFields: { ts: { $toDate: "$_id" } } },
    lastInserted(1),
  )

We get this output:

/*
@output

{
    _id: 60245842f36c37895594ebb0,
    actionCateg: 'fav',
    hw_id: 'distrail__01',
    user_id: '6024574e948c3b4a8cb99d2e',
    hw: 'distrail',
    isFav: true,
    ts: 2021-02-10T22:03:46.000Z
  }
*/

Notes

Let me read in natural English what the query is doing:

Return an array from the mongo collection,

but in the first stage of the pipeline, shape the resultset by adding a new field to it called ts (meaning timestamp, which will represent the original insert date of the record).

And the value of the new ts field will be retrieved from the _id field (the PK field).
Notice two things:

we use the mongo $toDate operator as a utility function that extracts the date from the _id field. Note the interesting syntax, which is when we want to reference a field key from the value side ({$MONGO_OPERATOR: VALUESIDE}), we prefix the key name with the dollar sign sigil. Since it's on the value side, it must be a string, so it's wrapped in quotes.

So to summarize, we added a field to our result set, whose key is "ts" and whose value is the extracted date from the _id field.

The output resultset has a new field called ts that is not in the document in the db collection. It was calculated on the fly.

This is a short example, but I wanted to explain it fully.

Not related to "dates" — but just to explain — the second stage (and last stage) is my lastInserted utility func which simply retrieves the last inserted doc from the collection.
Here's the source code for it if you're interested:

/*
@param {number} lim - the count of the most recent inserted docs we want
@return {object[]} - an arr of two stages
*/
export const lastInserted = lim => {
  return [
    sortDesc("_id"),
    limit(lim),
  ];
};

What's Next

As the Series continues we'll cover more interesting queries with dates.

For example, We can select sets of docs between dates, like monthly, weekly, daily, or hourly.

We can even create dimensions of time-series reports.

The limit is your imagination. :)

Delightful Database Design - Mastering MongoDB Queries - Series #11

Functional Javascript — Fri, 12 Feb 2021 16:40:43 +0000

Legend

AF = Aggregation Framework

Intro

We've covered a lot of powerful features of shaping data that we retrieve from the database, but we've also barely scratched the surface.

Other powerful data manipulation actions we can do:

joins (using the $lookup operator)
writing results to new collections (using the $out operator)
performing unions on dataset (using the $unionAll operator)
merging datasets (using the $merge operator)

These are all "stage operators". They are stages of our aggregation pipeline, which is our full query, and is represented as an arr of stages. Each stage is a single atomic unit of our database query.

Here is some pseudocode to understand the structure of composing the database query:

const aggregationPipeline = [
 { $stage1 },
 { $stage2 },
 { $stage3 },
]

Then we simply make the call to the database using the "aggregate" func.
The aggregate func takes one parameter, the aggregation pipline arr; which is simply an arr of objs called stages.
That's all there is to the "MongoDB Aggregation Framework" (AF) from the bird's eye view:

return await client.db(dbName).collection(collName).aggregate(aggregationPipeline).toArray();

Question and Answer

1.
How does AF compare with other database query systems, like Mongoose or GraphQL.

AF vs Mongoose:

The AF is well beyond the capability of Mongoose.
When you're composing AF queries, you're writing queries directly for the MongoDB engine and optimizer. It natively understands the query and optimizes it. AF is the future direction of the MongoDB company and ecosystem. They are pouring tens of thousands of development and research hours into it to make it better every iteration.

AF vs GraphQL:

These are two different products.
For example, your GraphQL resolvers can use the AF to retrieve data. GraphQL has it's set of usecases that it is designed for, like bringing various (or disparate) datastores together into a coherent and consistent orchestration.
AF is deceptively simple; But also deceptively powerful and performant. So unless your enterprise usecase calls for a GraphQL-style solution, you may be better off selecting the least amount of moving parts.

What's Next?

In the next few series of articles, we'll do some advanced and creative data-shaping activities. Taking the AF approach makes this possible.

Mastering The Database - Accessing Nested Records - Series #10

Functional Javascript — Thu, 11 Feb 2021 17:13:10 +0000

Intro

In the last series we retrieved a random sample of docs.
It was very simple. Essentially a oneliner.

But supposes we wanted to select a random sample of items that existed as subdocs of the outer level docs in our app?

Actually, it's not too hard.
And we don't use any loops.
Instead we "unwind", or your could think of it as "flattening" our arr of docs.

If an artist has an arr of albums, then each artist doc has an arr of subdocs.

If an artist's arr of albums each contain an arr of song, then we have a doc that has an arr of subdocs whose subdocs also have an arr of subdocs.

Here is what the outer doc's data shape look like:

/*
data shape:
 {
    "artist": "Julian Lennon",
    "albums": [
      {
        "albumTitle": "Valotte (1984)",
        "albumSongs": [
          {
            "song": "Valotte"
          },
          {
            "song": "O.K. for You"
          },
          {
            "song": "On the Phone"
//....
*/

Here is the query to select a random set of song (in this case 5 of them)

  mgArr(dbEnum.nlpdb, collEnum.songsColl,
    unwindArr("albums"),
    unwindArr("albums.albumSongs"),
    randomSample(5),
    projectIncludeNoId("artist", "albums.albumSongs.song"),
  )

Here are five random songs.
We've included the artist so we can see who the song is performed by:

/*
output:

[
  {
    "artist": "Miley Cyrus",
    "albums": {
      "albumSongs": {
        "song": "Girls Just Wanna Have Fun"
      }
    }
  },
  {
    "artist": "Creedence Clearwater Revival",
    "albums": {
      "albumSongs": {
        "song": "Penthouse Pauper"
      }
    }
  },
  {
    "artist": "Judas Priest",
    "albums": {
      "albumSongs": {
        "song": "Out In The Cold"
      }
    }
  },
  {
    "artist": "Akon",
    "albums": {
      "albumSongs": {
        "song": "Throw Dat"
      }
    }
  },
  {
    "artist": "Nazareth",
    "albums": {
      "albumSongs": {
        "song": "Hit The Fan"
      }
    }
  }
]
*/

Notes

1.
The unwindArr is a wrapper func around the $unwind stage operator.
The wrappers make the code look cleaner, and they're usually oneliners:

export const unwindArr = k => ({ $unwind: "$" + k });

2.
We have to unwind the outer arr, then we can next unwind the inner arr. We could keep on going if there were more arrs. For example, each song could have an arr of songwriters.

3.
Once all the songs are unwound, we can grab a random sample of them using the $sample stage operator.
The RandomSample wrapper is a simple oneliner:

export const randomSample = lim => ({ $sample: { size: lim } });

4.
Notice the dot notation in the unwind func in order to express the path to the nest arrs.

5.
Finally, for ease of display reason, I "project" out only two fields. My wrapper func, the "NoId" variant, excludes the Primary Key. I use this variant just for testing
The raw $project syntax would look like this (0 means exclude, 1 means include):

{ $project: { _id: 0, artist: 1, "albums.albumSongs.song": 1 } },

6.
Take note also, that conveniently, we can swap out and use either the wrapper funcs, or the raw MongoDB stage syntax with our call to the MongoDB aggregate func. This is because the wrapper funcs simply just returns the raw syntax anyhow, of each stage in the pipeline.

Resources

https://docs.mongodb.com/manual/reference/operator/aggregation/unwind

https://docs.mongodb.com/manual/reference/operator/aggregation/sample

Mastering The Database - Node.js, Express.js, MongoDB - Series #09

Functional Javascript — Thu, 11 Feb 2021 04:54:18 +0000

Intro

Let's retieve some random items from a database collection

app.post(apiEnum.api_find_artists_random, async (req, res) => {
  res.json(await mgArr(dbEnum.nlpdb, collEnum.songsColl,
    randomSample(50),
  ));
});

Notes

1.
We are using MongoDB's aggregation framework to retrieve a random sample of artists.

2.
This query has one aggregation stage, called "randomSample", which is just a functional wrapper around the $sample stage operator:

/**
@func
retrieve a random sample of docs

@param {number} lim
@return {object}
*/
export const randomSample = lim => ({ $sample: { size: lim } });

3.
Notice our query is already wrapped in an Express Route. Any registered client call that hits that route will retrieve the results.

Client Call

Here is our client call:

const fetchArtistsRandom = async () => {
 const r = await fetchArrNoParams(apiEnum.api_find_artists_random);
// ...
};

Client Call Notes

1.
Notice the custom fetch wrapper func.

2.
The "NoParams" portion of the name means we do not pass up any parameters to the server.

3.
the "fetchArr" means it returns an arr of results. (as opposed to just a single doc, which is called fetchObj).

4.
Notice we pass in only the URL segment of the API. This fetch wrapper func will retrieve the server base url. This url is set at build time. The build tool will modify the config file with the proper base url to the server.

5.
The fetch wrapper func will call axios using the POST method. All calls use the POST method to avoid the disadvantages associated with the passing of data over the URL.

6.
The fetchArtistsRandom event handler func can be called from the onMount lifecyle func, or from a wired up click event.
eg:

on:click={fetchArtistsRandom}

Example UI Display

What's next

Here we've shown an example querying data from an API Route, from the User Interface.
It takes just a matter of a minutes to set up a query for the User Interface once you have a set of tools in place.

If you have any questions, let me know.

Resources

Select random items using the $sample stage operator:
https://docs.mongodb.com/manual/reference/operator/aggregation/sample

MongoDB: Inserting Data - Series #08

Functional Javascript — Tue, 09 Feb 2021 17:43:43 +0000

Intro

In the last article in the series I showed how to establish a connection to MongoDB.

Here I'll show how to interact with data using that connection object.

Here's a simple insert:

/**
@func
insert a single doc into a coll

@param {string} dbName
@param {string} collName
@param {object} o - the doc to insert into the coll
@return {Promise<object> | null} the inserted doc, or null if exception
*/
const mgInsertOne = async (dbName, collName, o) => {
  const client = await mongoAsync();
  try {
    return await client.db(dbName).collection(collName).insertOne({ ts: getDateStandardWithSeconds(), ...o });
  } catch (err) {
    console.error(err);
    return null;
  } finally {
    if (client) {
      client.close();
    }
  }
};

export default mgInsertOne;

Here's an example usage:

 mgInsertOne("myCompanyDB", "userColl", { firstName: "fluffy" });

Here's the sample output:

/*
@output
{
  ts: '[2021-02-09 Tue 09:21:21]',
  firstName: 'fluffy',
  _id: 6022c491834ad44058181f5b
}
*/

Notes

The "mgInsertOne" utility func will return our inserted record. This is useful for delivering feedback in the UI.

Notice the utility func closes the client connection in the "finally" block

If the DB does not exists, it gets created dynamically.

If the collection does not exists, it gets created dynamically.

If you don't supply a primary key, which is field called, "_id", then the framework autogenerates one for you.

The client code of this func is a simple one liner. Wrap it in an Express.js route, push it to a provider like Heroku, and you're live within minutes — from composing the query — to having it available for clients to use.

What's Next?

If you have any questions let me know.

The best way to perform queries in Mongo is using their "aggregate" func. It takes an arr of objs. Each obj is a part of the query that we're composing. It's an extremely powerful command that allows to you to shape and reshape your data any way your wish.
We'll cover more real-life examples of this powerful tool.
Start reading from the beginning of this series to get an introduction to this "Aggregation Framework".

MongoDB: Connect to the Mongo Driver - Series #07

Functional Javascript — Mon, 08 Feb 2021 19:02:25 +0000

Intro

This is our lowest level code for establishing a connection to the Mongo database.

import { MongoClient } from "mongodb";
import env from "@root/mid-libs_node/bl/envs/processEnv";


const opt = {
  useNewUrlParser: true,
  useUnifiedTopology: true,
  connectTimeoutMS: 60000,
};


/*
@func
get a connection obj to atlas mongodb

@return {Promise<MongoClient> | null}
*/
const mongoAsync = async () => {
  try {
    return await MongoClient.connect(env.mongoUriAtlas, opt);
  } catch (err) {
    console.error("mongoAsync: CATCH: err.stack... ", err.stack);
    return null;
  }
};

export default mongoAsync;

Notes

1.
We import the npm library, which is the Mongo Driver software that allows us to establish a connection to Mongo

2.
We import our database credentials from the git-ignored ".env" file.

3.
We set our options.
The first two options removing warning flags.
I set the connection timeout a little higher to facilite the pushing of multi-megabyte catalogs and corpora (the plural of corpus) of data.

4.
This wrapper func returns the Mongo client object, which gives your code full power.

5.
The responsibility is on the client code to close the connection.
(I'll show the idiom for that later in the series)

6.
"Atlas" is the name of the MongoDb company's DBaaS (Database as a Service) offering.

7.
That's about it. It's really just a simple wrapper.

What's Next

If you have any questions let me know.

Next I'll show you a couple example of using this wrapper around the Mongo Driver; so you can write fast—sometimes single-line—database queries.

MongoDb - insert embedded documents - series #06

Functional Javascript — Sun, 07 Feb 2021 18:25:56 +0000

Intro

Let's say a user wants to send us some correspondence.

[aside]
Do we want to store it with the user doc?
Or store it in a doc in a separate collection?
That's the "referencing vs embedding" question.
There is a checklist of questions you can ask, and we'll cover that in depth later in the series.
The short and fast answer for this use case is that you could go either way, and it'll work fine.
[end aside]

In any case, embedding content is the power of NoSQL.
Below is a function that will do this.
First let's visualize the user data model:

/*
user data model

{
 _id: "",
 firstName: "",
 lastName: "",
 email: "",
 correspondence: [
  {text: "", email: "", ts: ""},
  {text: "", email: "", ts: ""},
 ]
}

*/

The Mongo utility func

See the notes below for an explanation

/**
@func
update a doc
- by pushing an obj onto one of it's fields that is of type arr

@notes
isUpdate = false - if no match is found for the outer doc, then a new outer doc is inserted
isUpdate = true - if a match is found for the outer doc

@param {string} dbName
@param {string} collName
@param {object} objFilter - match on this criteria
@param {object} objToUpdateOrInsert - update the doc at that location with this data
@return {Promise<object>} - the updated record, whether inserted or updated
*/
const mgUpsertOnePush = async (dbName, collName, objFilter, objToUpdateOrInsert) => {
  const client = await mongoAsync();
  try {
    const r = await client.db(dbName).collection(collName).findOneAndUpdate(objFilter, { $push: { ts: getDateStandardWithSeconds(), ...objToUpdateOrInsert } }, { upsert: true, returnOriginal: false });
    l("updatedExisting (isUpdate): ", r.lastErrorObject.updatedExisting);
    return r.value;
  } catch (err) {
    le("mgUpsertOnePush: CATCH: there was an error...\n", err.stack);
    return null;
  } finally {
    if (client) {
      client.close();
    }
  }
};

Notes

1.
The option, "upsert: true" is how we enable upserting in Mongo

2.
The option, "returnOriginal: false" forces Mongo to return back the doc AFTER it's been updated. We want this so we can display to the user a confirmation of what they just sent us.

3.
The Mongo command, "findOneAndUpdate" means only one doc will get modified, the first one found.

4.
The "$push" operator is the embedding power here. It pushes this doc onto one of the doc's fields that is of type arr.

5.
The log statement, that starts with "l("updatedExisting"", just tells us if an "insert" or "update" was performed. I.e. was a record found? Or is this a new record?

6.
The "le" func means "log erroro" and is just a wrapper func for "console.error"

Example Usage

app.post(apiEnum.api_upsert_user__featureRequest, async (req, res) => {
    const { user_id, email, name, text} = req.body;
  res.json(await mgUpsertOnePush(dbEnum.nlpdb, collEnum.users,
    { email },
    {
      correspondence: {
        user_id,
        name,
        type: "featureRequest",
        text,
        ts: getDateStandardWithSeconds()
      }
    }
  )
  );
});

Notes

1.
We set up an Express.js route to accept correspondence from all our client applications.

2.
The client submits four fields...
user_id
email
name
text

3.
The { email } argument of the call to mgUpsertOnePush is what it tries to match on. If it doesn't find a user doc with that email, it will insert a new user doc.
This is useful if you accept correspondence from users who are not signed up already to your site.

4.
The last argument will find find a field in the user doc called, "correspondence" (or create it if it's not there already, as an arr), and will push this sub-document (which is the correspondence content) onto the end of the arr.

Test it out

1.
Let's fill in the form:

2.
Let's submit the form:

3.
Confirm the DB entry:

lpromiseStringify(
  mgArr(dbEnum.nlpdb, collEnum.users,
    matchExact("email", "joe@sixpack.com")
  )
);

/*
@output
[
  {
    "_id": "60203a9ef36c378955b02a9d",
    "email": "joe@sixpack.com",
    "correspondence": [
      {
        "name": "Joe Sixpack",
        "type": "featureRequest",
        "text": "This is my inquiry",
        "ts": "[2021-02-07 Sun 11:08:13]"
      }
    ],
    "ts": [
      "[2021-02-07 Sun 11:08:14]"
    ]
  }
]
*/

What's Next?

If you have any questions, let me know.

You many notice that doing upserts, inserts, updates, delete does not involve the MongoDB Aggregation Framework (AF). However with the AF, we still have the power to insert or update its resultsets if we choose to. We don't always have to return the resultset to the client. This allows us to reshape data. We'll talk about that later in the series.

That's for reading. And let's let the journey continue! :)

NodeJS, MongoDB - "OR" queries - series #05

Functional Javascript — Sat, 06 Feb 2021 02:19:41 +0000

Intro

When querying the database in MongoDB, all "Matches" you perform are conjunctions (i.e. "AND" operations).

To perform disjunction queries (i.e. "OR" operations), I'll use a wrapper func (helper func) called "matchOr".

Here's an example. For an indepth explanation, see the Notes section below.

lpromise(
  mgArr(dbEnum.nlpdb, collEnum.users_actions,
    matchExact("type", "inquiry"),
    matchOr(existsNot("isDeleted"), isFalse("isDeleted")),
    lastInserted(5),
  )
);

Notes

USECASE Let's say users of our website are submitting inquiries. We want to have a look at the last few inquiries.
We want to exclude deleted inquiries. Deleted docs (records) are soft-deletes, meaning they are flagged as deleted (isDeleted=true) rather than actually hard deleted from the db collection.
The disjunct (the "OR") situation is that if there is no isDeleted field then the doc has never been deleted. But if there is an isDeleted field, only in cases where it's value is true is it deleted.
The matchExact("type", "inquiry") stage filters by only correspondence types that are inquiries (this is an "AND" filter).
The matchOr stage reads as this, "return the record if there is no isDeleted field, OR, return the record if the isDeleted field exists but its value is false". With this we are filtering out any "isDeleted=true"
The last stage just retrieves the last 5 inserted records.
The raw Mongo Aggregation Framework Stage syntax would look like this...

{ $match: { $or: [{ isDeleted: { $exists: false } }, { isDeleted: false }] } },

So we've saved ourselves a bunch of curlies. :)

What's Next

If you have any questions, let me know, and I'll try my best to answer them.
Other constructs in the example code here like "lpromise" and "mgArr" were explained earlier in this article series, so make sure you read those if you are new to the series.
Tomorrow I'll show an interesting but very common type of operation:
We want to update a document by pushing a document onto one of its fields that is of type array.

Res

https://docs.mongodb.com/manual/reference/operator/query/or/

NodeJS, ExpressJS, MongoDB - Paginate - series #04

Functional Javascript — Fri, 05 Feb 2021 02:49:50 +0000

Intro

A quick example on actually a very important feature: "paginate"

Always paginate your resultsets. This protects your system from accidental or malicious oversized resultsets being retrieved.

Pagination is very easy in MongoDB. See the notes section below.

app.post(apiEnum.api_find_artists__songRegex, async (req, res) => {
  let { searchTerm, page } = req.body;

  //#guard 1
  if (isNotBetween(page, 1, 500)) {
    page = 1; //defaultVal
  }

  //#guard 2
  if (isEmptyStrOrNil(searchTerm)) {
    return res.status(400).json([{ error: "the search term was empty" }]);
  }

  const regex = new RegExp(`${searchTerm}`, "i");
  res.json(await mgArr(dbEnum.nlpdb, collEnum.songsColl,
    copyField("searchResult", "albums"),
    unwindArr("searchResult"),
    unwindArr("searchResult.albumSongs"),
    matchRegex("searchResult.albumSongs.song", regex), //54
    paginate(50, page)
  ));
});

Notes

See series #03 for an explanation of some of these stages like "copyField" and "unwindArr". Here we'll concentrate on the one database query stage, "paginate".
The above Node.js Express router endpoint returns the paged results of a user search for a string of characters in a song.
The paginate wrapper func wraps the skip and limit funcs

/**
@func
limit a resultset to a particular requested page of results

@param {number} lim - page size
@param {number} page - page number to retrieve
@return {object[]}
*/
export const paginate = (lim, page) => {
  return [
    skip(lim * (page - 1)), // 50 * 2 gets results 51 to 100
    limit(lim),
  ];
};

The skip and limit funcs both wrap the MongoDB $skip and $limit pipeline stage operators

export const limit = lim => ({ $limit: lim });

export const skip = n => ({ $skip: n });

So paginate returns an arr of two stages because it uses two staging operators. You don't have think about that though.
You only have to call paginate and pass in two numbers.
An example of the resultset in the UI:

What's Next

If you have any questions let me know
We'll keep moving the needle forward with more enterprise patterns in the subsequent articles in this series