DEV Community: alex

MongoDB change streams (web SDK) for real time data tracking

alex — Sat, 11 Dec 2021 23:22:34 +0000

The real reason for this article is I'm writing down stuff I might refer to in future because I hate the MongoDB docs.

Skip intro.

Intro

In my on-going journey to become an evil data baron I've come to appreciate the value proposition of villainously tracking people's GPS coordinates in real time.

The problem is, if we make POST requests to update 10,000 people's coordinates every 30 seconds for 12 hours a day, we'd make already 864,000,000 calls. This is way above most rate limits. Costs would become prohibitive.

I'm not a top tier filthy rich villain machinating some big name tech mega-conglomerate. I'm just a dev, sitting at my desk, looking at Digital Ocean App Platform $5 droplets.

Fortunately, we can stream data. I've not calculated the costs in a detailed way but this seems potentially a lot cheaper.

Streaming is a very different paradigm. After a lot of research I settled on MongoDB change streams. Change streams allows us to monitor data in real time. Note: only monitoring.

It does not allow us to add or update it in real time. CRUD methods must still be done via regular HTTP calls.

That really disappointed me. I'd already spent quite some time reading its docs and then integrating it into my app. So now I'm going to rip it out and use an older strategy with a traditional self-hosted back end plus database for my current project. Sad. I really wanted to use a "serverless" DBAAS.

Nonetheless it's pretty useful. But, the documentation is byzantine. It's a kafkaesque monstrosity that gives me a literal neck ache to look at it. Documentation complexity is a kind of evil in itself.

So, the real reason for this article is I'm writing down stuff I might refer to in future.

Tech stack

TypeScript
React
MongoDB Realm

Setup

I assume for the most part that you're able to do setup.

Set up cluster, database, and/or collection

Create database and/or collection

Docs.

Usually, I'd set up the database and then let my front end generate the collections. But in this case, we need the collection to show up in the Realm app dashboard. So use the MongoDB UX to create it.

In this example, imagine that I've created a database called "users" and one collection in it also called "users".

Give your front end and/or back end client access

Click on "Network Access" in the sidebar of your dashboard and add it. Docs.

Do note that if you're connecting from a service like Vercel that has dynamic IP's you'll have to set IP to 0.0.0.0/0. Docs. I don't have an opinion on the security stuff here. I probably should.

Setup Realm App

Create Realm App

Go to your MongodDB dashboard, click on the Realm tab, and click on "Create a New App". Docs.

Allow user access

It might be important to note I'm writing mainly from the perspective of a front end client here.

It was easy to miss this. In order for users to connect, we have to:

Authenticate users, docs
- Click on "Authentication" under "Data Access" in the sidebar of Realm dashboard
- Enable the auth type you want and go from there
- Because I don't care about identifying my users, I use Anonymous Authentication
Set up rules
- Click on "Rules" under "Data Access" in the sidebar of Realm dashboard
- Set rules for users on your collection
- owner means the user/client who inserts the record, non-owner means anyone else

Front end

Basic setup

We'll use the Realm Web SDK with React.

Run npm i realm-web.

If you're using Anonymous Authentication for users/clients then the only identifier you need to access the Realm app is the Realm app ID. It should be right at the top of your Realm dashboard.

I've placed all my Realm-related methods in the same module and instantiated a Realm app instance right at the top. All the methods below it have access to this instance:

import * as Realm from "realm-web";

const realmApp: Realm.App = new Realm.App({ id: realmAppId });

I created a method to auth users/clients. It should be nice to put user into global state but I'm not showing it here:

export const realmLogin = async () => {
  const credentials = Realm.Credentials.anonymous();

  try {
    const user: Realm.User = await realmApp.logIn(credentials);

    if (user.id !== realmApp?.currentUser?.id) {
      // Handle auth fail
    }

    return user;
  } catch (err) {
    // Handle auth fail
  }
};

We can have users auth/log out even though we don't really have a reason to if users are anonymous. Here's how anyway: realmApp?.currentUser?.logOut()

Set up change stream watcher

I cribbed the following code directly from the Web SDK docs.

It's important to note that we're using the Web SDK, not the Node one. They are different. It's easy to start reading the Node docs and not realise for a while that we're reading the wrong docs.

I prefer the Node change stream patterns more. But, well, we're stuck with this one for now. Anyway, here's a method for watching the stream:

const watchCollection = async () => {
  // Connect to the users collection in the users database
  const mongodb = realmApp?.currentUser?.mongoClient("mongodb-atlas");
  const users = mongodb?.db("users").collection("users");

  if (!users) {
    return;
  }

  // This is the watcher, note the filter
  const changeStream = users.watch({
    filter: {
      operationType: "insert",
      "fullDocument.banned": false,
    },
  });

  for await (const change of changeStream) {
    switch (change.operationType) {
      case "insert": {
        const { documentKey, fullDocument } = change;
        console.log(`new document with _id: ${documentKey}`, fullDocument);
        break;
      }
      case "update": {
        const { documentKey, fullDocument } = change;
        console.log(`updated document: ${documentKey}`, fullDocument);
        break;
      }
      case "replace": {
        const { documentKey, fullDocument } = change;
        console.log(`replaced document: ${documentKey}`, fullDocument);
        break;
      }
      case "delete": {
        const { documentKey } = change;
        console.log(`deleted document: ${documentKey}`);
        break;
      }
    }
  }

  return changeStream;
};

List of available change events.

The filter

The cool thing that attracted me to MongoDB Realm's change streams is the filter. I was so excited about it.

A lot of other databases might allow monitoring in real time, but they don't allow queries in real time on the stream. Change streams allows it.

I think Apache Kafka's interactive queries do the same thing but it's only available in Java and not in their JavaScript SDK. It's why I picked MongoDB's Realm change streams instead.

Notice that we're filtering for two criteria:

filter: {
      operationType: "insert",
      "fullDocument.banned": false,
    }

Because of this, the Realm app will send to the client only records that have been inserted, and where the record field does not contain {"banned": true}.

The filters can probably be a lot more complex, involving mathematical operators and the like, just as we could do on regular MongoDB queries. It's just a simple example here.

Perhaps important to note is that this filter is passed on to the Realm app or kept in cookies or something (I think? Look, I didn't excavate the source code here, ok?) when our front end client connects. So if we're working in a framework that hot reloads like React, we cannot expect changes to the Realm methods to take effect just because our front end app has recompiled. We have to reload the page in the browser as well.

Get the party started

With the main methods written, all we have to do now is:

realmLogin();
watchCollection();

Functions, HTTPS endpoints, other features

Realms has some other pretty neat features, functions, triggers and HTTPS endpoints being just a few of them.

Here's one of my functions that I added to "Functions", under "Build", in the Realm sidebar. I named it "deleteAllUserData":

exports = async function(){  
  const userid = context.user.id;

  const userColl = context.services.get("mongodb-atlas").db("users").collection("users");
  const res = await userColl.deleteMany({userid});

  return res;
};

Using the name, I call it in my React front end like this:

user.functions.callFunction("deleteAllUserData")

It deletes all the records created by the current connected user/client instance.

End

That's it. Now, get outta here and wreak profit.

Checklist to estimate time to make a thing

alex — Fri, 16 Jul 2021 08:13:26 +0000

Writing this (mainly for myself) because we keep getting bit in the butt with our estimations.

This checklist helps interrogate our assumptions. On first glance a task may seem easy but it could hide a ton of other tasks.

Generally I find quintupling our first estimate brings us a lot closer to the truth.

Data

Where does it come from?
- Is it easy to access?
- Do we need authentication?
Does it contain all the properties we need?
Have we actually seen it or are we just assuming that we probably have this data because it's so obvious? (Don't assume. lol. 😭)
Does it need to be transformed?
- Is it complex to transform?
- Does the data's reliability depend on many third parties? E.g., country calling codes 📞 could start with +, 0, 00, or a whole slew of other patterns depending on country. It might take a lot of investigation just to figure out what the data is really about, and how to break it up into component parts suitable for our use case.
- Does the new form need to be stored somewhere?
- Should we work it into a regular ETL and CD process? E.g., daily cron job to ETL public transport traffic data, transform it, test it, then deploy to a server that uses Docker in Kubernetes in Azure 😅.
Do we need to communicate with several parties over a period of time to get this data, and in a form we can use?

UI/UX (if front end)

What does it look like on mobile and desktop?
How should the UX behave?
Are the updated designs readily available?
How much time to add accessibility?
- E.g., invisible elements that screen readers should read out, can tab through all elements, UX for focused elements.
Does it work on all browsers we support?
Do other teams' CSS override ours in production? Could happen when multiple teams develop separately but their work is merged in prod.
Tests!

Reused code/components

Do we have to reuse code?
- How easy is it to edit and/or inherit for the new use case?
Is it legacy code?
- Do we have to update libraries? If yes, do we need regression testing?
Is it complex?
- E.g., deeply nested modules or 1000+ lines in the module.
Will regression testing be complex? E.g., if edited code touches 10 services, manual testing might be required in spite of extensive test coverage, especially when it's UI/UX.

Tests

TESTS. (I'm not great at this. 🤷🏻‍♀️)

Could add like 30-40% more time but totally worth it.

Meetings

Some days I code for like 30 min a day because of meetings. But the task in the agile scrum kanban tomato thingamajig backlog says the task I'm working on should take 2 hours. So then I end up spending maybe 4 days on the thing in total, because of meetings and complexities mentioned above. Not even exaggerating.

Buffer time

We're human. Procrastination, social obligations, and laundry are real.

And horror of horrors, equipment breaks down, WiFi fizzles out, botnets attack... 🙀

Build this in.

Workflow

All of this assumes that we already have in place a smooth company/department-wide workflow where business/commercial 📊 side snazzily breaks the product up into features, user stories, and/or tasks for easy bite-size dev consumption. Otherwise, that's a whole other thing.

Invisible content for screen readers with aria-labelledby

alex — Mon, 12 Jul 2021 16:14:33 +0000

This is more a note for myself than anybody else.

Problem

What if you have a group like the following but for some reason you're not allowed to have a title to tell people what it's for? Maybe your designers think it's contextual enough for sighted people or something.

<div role="radiogroup">
  <input type="radio" id="fish" name="fish">
  <label for="fish">🐟</label>

  <input type="radio" id="mammal" name="mammal">
  <label for="mammal">🧸</label>
</div>

Solution

WebAIM has really good tips on how to hide content from sighted users. I want to focus only on what they say about display: none and visibility: hidden here:

These styles will hide content from all users. The content is removed from the visual flow of the page and is ignored by screen readers. Do not use this CSS if you want the content to be read by a screen reader. But DO use it for content you want hidden from all users.

True. However, Accessibility Developer Guide says:

elements hidden using CSS can still be referenced

How? With aria-labelledby.

Demo

Let do this with our radiogroup example:

<div role="radiogroup" aria-labelledby="hidden-content">

  // Exciting new span!
  <span id="hidden-content">Which animal family do you like better?</span>

  <input type="radio" id="fish" name="fish">
  <label for="fish">🐟</label>

  <input type="radio" id="mammal" name="mammal">
  <label for="mammal">🧸</label>
</div>

The CSS:

#hidden-content {
  display: none;
}

This way, the span is hidden from sighted users but can be read out by screen-readers.

Why not `aria-label`?

We might ask, why not just apply aria-label to the wrapper div? ADG says: "Text added with aria-label is not searchable in browsers."

The end

🙂

Empty emails from specific address from gmail trash

alex — Mon, 15 Jun 2020 01:45:23 +0000

Problem

One of my clients uses Gmail and their abuser keeps emailing them. They read them and get affected.

Gmail allows us to filter incoming emails to spam or trash. They've done that. But they go and read them in trash (emotions are hard).

Gmail doesn't have a filter to let us empty emails from specific senders from trash.

Solution

A Google script that automatically empties abuser's email from Gmail trash.

Notes

Skip to end of article for full script if that's all you need.

This script only empties emails from specific sender from trash. It does not filter incoming ones to trash. To do that, use Gmail's filters. Remember to set "never send to spam" so the emails always go to trash.

Some have pointed out that emails might be evidence. Think about that before using this script. I cobbled this script together because we're prioritising mental health for now.

APIs

This solution uses two main APIs. I don't know their exact names but:

Gmail.

GmailApp (Apps-Script).

Code walkthrough

The script has to run from the client's Google account. We can test it on our own first.

First, log into our Google account. Then go to script.google.com. This is where we type in the following code.

Some prep

Take a glance at the preparatory stuff before I explain them:

const clientGmail = 'myClient@This-Has-To-Be-Gmail-Here.com'

const page = {
    q: "in:trash from:abuser@Whatever-Their-Email-Is.com",
    pageToken: null,
    }

const limit = 5400000

Client's email in const clientGmail. Note: the script has to run from the client's Google account, so this has to be their Gmail address. However, because we're testing this on our own accounts first, set this to our own Gmail address.

page object contains details in its q property that allows Gmail to search for the abuser's email in gmail trash.

const limit is the time period from which we want to delete the abuser's email. I've set it to 1.5 hours in milliseconds. This means the script deletes any email from the abuser that came in within the last 1.5 hours.

The function itself

Take a brief look and I'll explain:

function deleteFromTrash() {
  const threadList = Gmail.Users.Threads.list('me', page);

  threadList.threads.forEach(thread => {
  const id = thread.id
  const threadByOtherAPI = GmailApp.getThreadById(id)

  const threadDate = threadByOtherAPI.getLastMessageDate()
  const timeNow = new Date
  const period = timeNow - threadDate
  const recent = period < limit

  if (recent) {
    Gmail.Users.Threads.remove(clientGmail, id);
    }
  })
}

threadList uses the Gmail API to find all the threads that fit the description in page.q:

const threadList = Gmail.Users.Threads.list('me', page);

To clarify: threadList contains multiple threads. A thread contains one or more emails.

We then loop through threadList with a forEach loop:

threadList.threads.forEach(thread => {
    ...
}

As far as I know, the Gmail API doesn't let us access the thread itself, but it gives us its ID. So, during each loop, we use the GmailApp (Apps-Script) API to get the thread by its ID. Like this:

const id = thread.id
const threadByOtherAPI = GmailApp.getThreadById(id)

Next, we get the date of the latest email in the thread, and check if it's within our time period. We do this because emails from long ago are of sentimental or other value to our client, so we don't want to delete them.

Here's how we check:

const threadDate = threadByOtherAPI.getLastMessageDate()
const timeNow = new Date
const period = timeNow - threadDate
const recent = period < limit

Finally, if the email is definitely recent, we empty it from trash using the Gmail API:

if (recent) {
  Gmail.Users.Threads.remove(clientGmail, id);
  }

Turn on Gmail in Google Services

The script won't work just yet. In script.google.com where we've just written the script:

Go to "Resources" in the menu
Click on "Advanced Google services"
Scroll down to Gmail API and turn it on

Set project trigger

Final step here. From script.google.com:

Go to "Edit" in the menu
Click on "Current project's triggers"
In the new window that opens up, click the "Add Trigger" button in the lower right of the screen
Set the interval during which the script should run. Note that we've set const limit to 1.5 hours. My own interval is "every hour". So the script runs every hour and checks if any abusers from the abuser arrived in the last 1.5 hours.

The last point bears some thinking. I will do this in the next section.

Rate limits

Rate limits for scripts and their triggers can be found here. The problem is, it's not clear to me if we should be looking at "Current quotas" or "Current limitations".

I also don't know what the difference is between Consumer, G Suite Basic, and G Suite free edition. Is our regular gmail account the first, second, or third? No idea.

Anyway, I've far exceeded trigger limits in both tables by now. So maybe they don't matter? I have no flippin' clue.

But it might matter to you, for whatever reason. So, keep these limits in mind.

Full script

Remember to run this script and enable Gmail in Google Services on client's account.

const clientGmail = 'myClient@This-Has-To-Be-Gmail-Here.com'

const page = {
    q: "in:trash from:abuser@Whatever-Their-Email-Is.com",
    pageToken: null,
    }

const limit = 5400000

function deleteFromTrash() {
  let threadList = Gmail.Users.Threads.list('me', page);

  threadList.threads.forEach(thread => {
  const id = thread.id
  const threadByOtherAPI = GmailApp.getThreadById(id)

  const threadDate = threadByOtherAPI.getLastMessageDate()
  const timeNow = new Date
  const period = timeNow - threadDate
  const recent = period < limit

  if (recent) {
    Gmail.Users.Threads.remove(clientGmail, id);
    }
  })
}

Basic space setup and CRUD with Contentful CLI for development

alex — Sat, 08 Feb 2020 17:18:17 +0000

We love CLI's. They help us automate things. But Contentful's CLI documentation is scattered all over the place. Here, I walk you through the basic bits needed for development. This is not comprehensive, but it should help get you going.

Basic setup

Get the CLI

Brew, yarn, or npm. Eg: npm install -g contentful-cli

Ref: Contentful CLI git repo.

Get your CMA

Log into Contentful. Go to Settings. Go to Content management tokens. Click on "Generate personal token". Copy the token somewhere (important because you'll never see it on the site ever again).

Login on CLI

Run this in shell: contentful login --management-token YOUR_CONTENT_MANAGEMENT_TOKEN. It'll tell you it saved your access token in .contentfulrc.json in your present working directory.

Ref: Contentful CLI - Documentation

Create a space

contentful space create --name 'Your Space Name'. You might need --management-token and pass in your content management key. And --default-locale if you want to set a default locale that isn't us-En.

If successful, it'll show you your space name alongside your space-id. Copy the space-id down somewhere.

Ref: Contentful CLI - Space command

Remember your space

contentful space use -s YOUR_SPACE_ID. This lets the CLI remember which space you're working in so you don't have to pass in --space-id all the time.

Ref: Contentful CLI - space use command

Logout

Don't logout now. But if you want to: contentful logout. It'll delete the .contentfulrc.json.

Set up space with migrations

Now it's time to create content types (like database tables) and their fields in your space. The CLI uses a migration script written in JavaScript to do this. It's not immediately clear from the docs if we can write this in other languages and I don't have time to check right now.

In general, most of the info is here: contentful-migration - content model migration tool. But this is a large chunk of everything.

You'll need to look at the data model too, to figure out what you can use to make, say, a blog post title and a post body: Contentful data model

Here is a more specific way to set things up on a basic level: The right way to migrate your content using the Contentful Migration CLI

In brief, your script should look something like this, taken directly from the link above:

module.exports = function(migration) {
  const author = migration.createContentType('author')
  author
    .name('Author')
    .description('Author of a blog post')

  author
    .createField('fullName')
    .name('Full name')
    .type('Symbol')
    .required(true)

  author
    .createField('twitter')
    .name('Twitter')
    .type('Symbol')
    .validations([
      { "unique": true },
      { "regexp": 
        { "pattern": "^\\w[\\w.-]*@([\\w-]+\\.)+[\\w-]+$" }
      }
    ])

  const blogPost = migration.editContentType('blogPost')
  blogPost.createField('author')
    .name('Author')
    .type('Link')
    .linkType('Entry')
    .validations([
      { "linkContentType": ["author"] }
    ])
};

Put it in migrations/01-add-author.js and run contentful space migration migrations/01-add-author.js.

If you didn't run contentful space use -s YOUR_SPACE_ID earlier, you might have to pass in --space-id YOUR_SPACE_ID too.

I'll paste my whole blog setup migration script at the end of this article so you can have something a little meatier to look at.

Delete content types and fields

This is our development phase so we CRUD everything all the time. We've created content types. But how to delete? It's probably possible to use native commands to delete stuff but, again, it's not immediately clear from the docs how to do this.

I'm in a rush and I'm too tired from reading docs all day so I'll cheat a little and use this library called contentful-clean-space. Install it: npm install -g contentful-clean-space.

Then, run: contentful-clean-space --space-id YOUR_SPACE_ID --accesstoken YOUR_CONTENT_MANAGEMENT_TOKEN --content-types. This deletes all entries/records and content types too. Check out their repo for a bit more info.

(Reads of entries are through the Contentful CDA and updates are done later through the CMA)

The end

I think that's it.

References

CLI docs: basic use of CLI like login and how to do a migration. But does not include what a migration file looks like, and how to add content types and fields to them.

CLI migration docs: details on how to write the migration script.

Data model: tells you about what fields can be added to your migration script, but has no info on what the script itself looks like.

Automated Contentful migrations: a couple of examples for migration scripts worth skimming through for a start.

The right way to migrate your content using the Contentful Migration CLI: more useful details for migration scripts, like how to link from one content type to another:

const blogPost = migration.editContentType('blogPost')
  blogPost.createField('author')
    .name('Author')
    .type('Link')
    .linkType('Entry')

    // Isolates link to only the "author" content type.
    .validations([
      { "linkContentType": ["author"] }
    ])
};

How to do validations on items({}). Basically:

blogPost.createField("categories")
    .name("Categories")
    .required(false)
    .type('Array')
    .items({
        type: 'Link',
        linkType: "Entry",

        // Right here.
        validations: [{
            linkContentType: [
                "categories"
            ],
        }]

    })

Contentful Migration Cheat Sheet: clues you in on how to do some of these fields, and also nice reminders.

Delete content types and fields from CLI: contentful-clean-space

My blog site migration script

It's not the best but it's a good start. It sets up content types and fields for generic pages, blog posts, and default site settings.

module.exports = (migration, context) => {

    // BLOG POST CONTENT TYPE

    const blogPost = migration.createContentType("blogPost")
        .name("Blog Post")
        .description("Blog post model")
        .displayField("title")

    blogPost.createField("title")
        .name("Title")
        .type("Symbol")
        .required(false)

    blogPost.createField("body")
        .name("Body")
        .type("Text")
        .required(false)

    blogPost.createField("author")
        .name("Author name")
        .type("Symbol")
        .required(false)

    blogPost.createField("datetime")
        .name("Datetime")
        .type("Date")
        .required(false)

    blogPost.createField("categories")
        .name("Categories")
        .type('Array')
        .items({
            type: 'Link',
            linkType: "Entry",
            validations: [{
                linkContentType: [
                    "categories"
                ],
            }]
        })
        .required(false)

    blogPost.createField("tags")
        .name("Tags")
        .type("Array")
        .items({ "type": "Symbol" })


    blogPost.createField("featuredImage")
        .name("Featured image")
        .type("Link")
        .linkType("Asset")
        .required(false)


    // CATEGORIES CONTENT TYPE

    const categories = migration.createContentType('categories')
        .name('Categories')
        .description('Categories for blog posts')
        .displayField("category")

    categories.createField('category')
        .name('Category')
        .type('Symbol')
        .required(true)

    categories.createField('slug')
        .name('URL Slug')
        .type('Symbol')
        .validations([{ "unique": true }])
        .required(false)

    categories.createField('featuredImage')
        .name('Featured image')
        .type('Link')
        .linkType('Asset')
        .required(false)

    categories.createField('description')
        .name('Description')
        .type('Text')
        .required(false)


    // PAGE CONTENT TYPE

    const page = migration.createContentType("page")
        .name("Page")
        .description("Page model")
        .displayField("title")

    page.createField("title")
        .name("Title")
        .type("Symbol")
        .required(false)

    page.createField("body")
        .name("Body")
        .type("Text")
        .required(false)

    page.createField("featuredImage")
        .name("Featured image")
        .type("Link")
        .linkType("Asset")
        .required(false)

    // SITE SETTINGS

    const siteSettings = migration.createContentType("siteSettings")
        .name("Site settings")
        .description("Site Settings model")
        .displayField("siteName")

    siteSettings.createField("siteName")
        .name("Site name")
        .type("Symbol")
        .required(false)

    siteSettings.createField("author")
        .name("Author")
        .type("Symbol")
        .required(false)

    siteSettings.createField("address")
        .name("Address")
        .type("Symbol")
        .required(false)

    siteSettings.createField("phoneNumber")
        .name("Phone number")
        .type("Symbol")
        .required(false)

    siteSettings.createField("email")
        .name("Email")
        .type("Symbol")
        .required(false)

    siteSettings.createField("facebookLink")
        .name("Facebook link")
        .type("Symbol")
        .required(false)

    siteSettings.createField("twitterLink")
        .name("Twitter link")
        .type("Symbol")
        .required(false)

    siteSettings.createField("instagramLink")
        .name("Instagram link")
        .type("Symbol")
        .required(false)

    siteSettings.createField("defaultImage")
        .name("Default Image")
        .type("Link")
        .linkType("Asset")
        .required(false)
}

How to make URL path prefix for section of website in Gatsby

alex — Fri, 10 Jan 2020 12:24:51 +0000

Let's say we want all our Gatsby blog posts to have a /blog/ path prefix. Like: example.com/blog/20202-01-10-i-went-to-mars.

It can be any other section but let's go with blog for this article.

Most tutorials say to put the blog posts in src/pages/blog. The post URLs will automatically come out the way we want. However, that assumes we write each blog post as a JSX component and with its own graphql query if there's markdown involved. What if we use a page template for the blog posts? It won't work anymore. Didn't for me. (Update 24 Feb 2020: it does work, I was being stupid somehow back then, I think. But this article presents another valid way to do it anyway.)

Instead of doing that, let's put blogpostTemplate.js in src/templates/blogpostTemplate and each blog post in src/blogposts/individual_post_dir/post.md. Like this:

src/templates/
└── blogpostTemplate.js

src/blogposts
└── 2020-01-01-individual-post-dir
    ├── 2020-01-01-first-post.md
    └── shoe-800x600.jpg

I won't discuss how to make a page template. It's easy to find resources on that. Anyway, presumably you're here because you've already made a template and found path prefixes broken. I'll focus only on creating the path prefix.

Usual page template setup in gatsby-node.js

Now, open up gatsby-node.js in the project root directory. If you've already created a page template for the blog (or whatever other section of your site), the following code will be familiar to you.

There are two or three ways to do it. Some use async/await, some use then. I feel it's more a matter of aesthetic taste than anything else. I prefer async/await but for goodness only knows what reason I went with then for this project:

const path = require("path")
const { createFilePath } = require(`gatsby-source-filesystem`)

exports.createPages = ({ actions, graphql }) => {
  const { createPage } = actions

  return graphql(`

      blogposts: allMarkdownRemark(
        filter: { fileAbsolutePath: { glob: "**/src/blogposts/*/*.md" } }
      ) {
        edges {
          node {
            html
            frontmatter {
              path
            }
            fields {   // Take note of this part
              slug
            }
          }
        }
      }
    }
  `).then(res => {
    if (res.errors) {
      return Promise.reject(res.errors)
    }

    res.data.blogposts.edges.forEach(({ node }) => {
        createPage({
          path: node.fields.slug,  // Look, it's here again
          component: path.resolve("src/templates/blogpostTemplate.js"),
          context: {
            slug: node.fields.slug,  // We'll create it in a bit
          },
        })
      })

  })
}

The main portions in the code above are:

const { createPage } = actions: destructure createPage from actions
return an allMarkdownRemark graphql query named blogposts
use createPage and the results in res.data.blogposts.edges.forEach() to create pages for each individual blog post using blogpostTemplate.

All pretty standard as a hundred and one tutorials out there will tell you. fields { slug } is important. It doesn't make sense now. But we'll make it in the next section. Gatsby will use this to create the path prefix.

The path prefix

Next, the exciting part. We create the path prefix. Right below the above code in gatsby-node.js, we paste in this:

exports.onCreateNode = ({ node, getNode, actions }) => {
  const { createNodeField } = actions
  if (
    node.internal.type === `MarkdownRemark` &&
    node.fileAbsolutePath.includes("blogposts")
  ) {
    const slug = createFilePath({ node, getNode, basePath: `src/blogposts` })
    createNodeField({
      node,
      name: `slug`,
      value: `/blog${slug}`, // Here we are, the path prefix
    })
  }
}

Whenever we run Gatsby develop or build, Gatsby creates nodes for all our directories and markdown files. It then uses these nodes for routing and linking and whatever.

As the name of the function above suggests, onCreateNode does something upon the creation of a node. We can do whatever we want with it. For our purposes, we have an if statement that checks if the file is markdown, and if it's contained in the blogposts directory.

If it is, then we use createFilePath to create a slug. The slug is just the blog post directory concatenated with the blog post filename.

To add the /blog/ path prefix, add it to value in createNodeField as shown above: /blog${slug}.

Gatsby will now use this value to create the paths to all our blogposts.

Following the example in the directory tree above, our post will be at example.com/blog/2020-01-01-individual-post-dir/2020-01-01-first-post.

Adapting the blog post template

Remember to edit blogpostTemplate.js. Tutorials usually tell us to have this query or very similar in it:

export const query = graphql`
  query($path: String!) {
    markdownRemark(frontmatter: { path: { eq: $path } }) {
      html
      frontmatter {
        title
        ...
      }
    }
  }
`

It looks for a path tag in the markdown frontmatter. However, we don't have path anymore. We have slug.

If you're already using slug, lucky you. If you aren't, change it:

export const query = graphql`
  query($slug: String!) {
    markdownRemark(fields: { slug: { eq: $slug } }) {
      ...
    }
  }

The end

For some reason, it's been frightfully difficult to find resources on how to do this. The best resource so far is actually the official Gatsby tutorial part 7. I think it's worthwhile to at least skim it for a better sense of things. In fact, I'm sure if people went through it, they'd figure things out faster than I did. But, I think, because the tutorial doesn't explicitly talk about prefixes, most people skip it.

A whole lot of people are asking about this online and trying all kinds of funny code acrobatics so I thought to write this down.

I may have misconceptions about how things are done in Gatsby. Or I've got things wrong. Or maybe you have a quicker, or more efficient way of doing this. Feel free to tell me these things.

How to use markdown in pages in Gatsby

alex — Mon, 07 Oct 2019 07:54:00 +0000

Most tutorials show how to create full pages with Gatsby's createPages, like for blogs. However, what if we don't want to create pages? What if we want to take markdown from markdown files and insert them into pages instead? That is, how do we insert markdown into JSX? I haven't found a lot of details on this. So, this is a fairly quick piece on how to do it.

In this article, we will display markdown from a markdown file in the default index.js landing page.

I kind of wrote this from memory. Please let me know if I made a mistake, missed anything, or if you have a faster method.

Why put markdown in pages?

I work with non-tech people. Sometimes, we want an SSG instead of a CMS. Markdown is easy to learn and write with.

This lets content folks publish with ease. They can upload markdown files and edit them whenever they like.

Basic setup

Create a new project

Set up the default starter project with gatsby new myproj. You can read more about starting a project here.

Install plugins

We need to install gatsby-source-filesystem to help Gatsby access the markdown files that we'll soon create. So, do this in your project root directory: npm install gatsby-source-filesystem.

Tell Gatsby about this new plugin by adding this to the gatsby-config.js file:

plugins: [
  {
    resolve: `gatsby-source-filesystem`,
    options: {
      name: `markdown-pages`,
      path: `${__dirname}/src/markdown-pages`,
    },
  },
]

Next, install gatsby-transformer-remark, which lets Gatsby use markdown properly: npm install gatsby-transformer-remark

Add it to gatsby-config.js:

plugins: [
  {
    resolve: `gatsby-source-filesystem`,
    options: {
      path: `${__dirname}/src/markdown-pages`,
      name: `markdown-pages`,
    },
  },
  `gatsby-transformer-remark`,
]

Note the path. It's ${__dirname}/src/markdown-pages. This means Gatsby should look into /src/markdown-pages for markdown files. Create the markdown-pages directory in src.

Make a markdown file

Create a markdown file in /src/markdown-pages. For our current demo purpose, let's create one called home.md.

Examples online usually tell us to create something like this:

---
path: "/"
date: "2019-10-06"
title: "About"
---

# About us

Here is the body of our content. It will be magically turned into HTML.

The part in --- is the frontmatter. We place metadata in there.

Graphql code

We'll make this really simple. We'll place our code in index.js so everything loads on the landing page. Find it in /src/pages/index.js.

We use graphql to look for the markdown file /src/markdown-pages/home.md. After checking out a handful of solutions, such as here and here, I decided that the allMarkdownRemark method is probably easiest. Feel free to let me know if you have an even easier option.

I haven't figured out how to fully properly use regex in graphql yet but we'll use it as shown below. It should work for files named in simple ways.

Here's the query to get the file:

graphql`
query {
  allMarkdownRemark(
    filter: {fileAbsolutePath: {regex: "/home/"}}
  ) {
      ...
    }
}
`

In the code above, the regex bit looks for file names with "home" in it. You might want to tailor the regex to your needs. It's currently fine for me because I'm using it for a simple site with just Home, Contact, and About pages.

Here's how to get the markdown content in home.md:

graphql`
query {
  allMarkdownRemark(
    filter: {fileAbsolutePath: {regex: "/home/"}}
  ) {
      edges {
        node {
          frontmatter {
            title
            date(formatString: "DD MMMM YYYY")
          }
          html
        }
      }
    }
}

In the code above, we've extracted a couple of frontmatter metadata. We've left out path because it's not needed today. We've also taken the html content.

It's not done yet though. We have to export it. So, the whole code looks like this:

export const query = graphql`
query {
  allMarkdownRemark(
    filter: {fileAbsolutePath: {regex: "/home/"}}
  ) {
      edges {
        node {
          frontmatter {
            title
            date(formatString: "DD MMMM YYYY")
          }
          html
        }
      }
    }
}
`

Place it just above export default IndexPage.

Query response is in props

The data from the graphql query will be in props. So, let's put it into IndexPage with { data }:

const IndexPage = ({ data }) => (
  ...
)

Let's also pass into IndexPage a method we'll use to read and display the markdown properly. We'll call it md().

To make things simpler, let's get rid of almost all the default JSX in IndexPage and just have {md({ data })} in it:

const IndexPage = ({ data }) => (
  <Layout>
    {md({ data })}
  </Layout >
)

Feel free to add other JSX to it if you like.

Method to read markdown

Before we continue, let's note that the structure of the data follows the graphql query we made above. After some exploration, I found out that allMarkdownRemark returns an array. I then used both the graphql playground and console in developer tools to navigate the structure properly. Despite that, it still takes some experimentation to get right.

In the interest of time, I'll just tell you the markdown content we want is in data.data.allMarkdownRemark.edges[0].node.

Now, let's build the function md(). It has two important parts:

First, we destructure frontmatter and html from the props.

Second, we return the JSX we want with the markdown properly rendered.

Here's how it looks:

const md = (data) => {
  const { frontmatter, html } = data.data.allMarkdownRemark.edges[0].node

  return (<div>
    <h1>{frontmatter.title}</h1>
    <p>{frontmatter.date}</p>
    <div
      dangerouslySetInnerHTML={{ __html: html }}
    />
  </div>)
}

The whole index.js

By now, our code should look like this:

import React from "react"
import { graphql } from "gatsby"

import Layout from "../components/layout"

const md = (data) => {
  const { frontmatter, html } = data.data.allMarkdownRemark.edges[0].node
  console.log(frontmatter.title)
  return (<div>
    <h1>{frontmatter.title}</h1>
    <p>{frontmatter.date}</p>
    <div
      dangerouslySetInnerHTML={{ __html: html }}
    />
  </div>)
}

const IndexPage = ({ data }) => (
  <Layout>
    {md({ data })}
  </Layout >
)

export const query = graphql`
query {
  allMarkdownRemark(
    filter: {fileAbsolutePath: {regex: "/home/"}}
  ) {
      edges {
        node {
          frontmatter {
            title
            date(formatString: "DD MMMM YYYY")
          }
          html
        }
      }
    }
}
`

export default IndexPage

Run gatsby develop and check out the page. It should be at localhost:8000.

Starting with saleor-storefront

alex — Fri, 20 Sep 2019 15:54:13 +0000

This is to document how I got started with saleor-storefront 0.6.0. This is saleor 0.0.0's front end made with React.

In this small walkthrough, I talk about a bunch of gotchas I encountered while trying to deploy them both. I hope this makes getting started with saleor a wee bit easier.

Most of the material online suggests that the writers are running both storefront and saleor on the same machine. I could not get this to work due to a few issues, including that the storefront won't run as described here by AbdulWaheedPasha.

Before AbdulWaheedPasha let me know about this issue, I thought it was because my many projects were putting too much strain on my Ubuntu instance. So, I decided to lighten the load a little bit and deploy the storefront to Netlify instead.

I start from the point just after we've cloned the saleor repo into our Ubuntu machine, and we've either forked or pushed the saleor-storefront to our own Git repo.

However, before deploying to Netlify, we have to set ALLOWED_HOSTS properly. This is done on the saleor site of things.

Set ALLOWED_HOSTS to *

I want to deploy saleor to https://example.com on my Ubuntu instance and saleor-storefront to https://storefront.example.com on Netlify.

In this case, because I run saleor by executing docker-compose up (sometimes with -d), I decided to add ENV ALLOWED_HOSTS 'storefront.example.com' to saleor/Dockerfile:

EXPOSE 8000
ENV PORT 8000
ENV PYTHONUNBUFFERED 1
ENV PROCESSES 4
ENV ALLOWED_HOSTS 'storefront.example.com'

This did not work. Storefront kept returning "Invalid host header" or otherwise just a blank page sometimes. So, I tried ENV ALLOWED_HOSTS 'https://storefront.example.com' instead. Did not work.

Most solutions online say to set it to *, which I felt might be a bit of a security issue because it would allow anyone to take data from the saleor back end. I went with it anyway, seeing how I couldn't solve the problem with a specific URL. It now looks like this: ENV ALLOWED_HOSTS '*'.

Before we go on, remember to run commands like npm run build-assets and npm run build-emails in accordance with the docs.

You should also run python3 manage.py migrate and python3 manage.py createsuperuser to make an admin for your site. If you also chose to use docker-compose up like I did, you can enter the saleor container with docker exec -it saleor_web_1 bash. Then run the migrate and createsuperuser commands.

Next, we turn to the saleor-storefront side of things.

Deploying saleor-storefront to Netlify

This should be pretty straightforward, I think. Follow Netlify's instructions to deploy repos. I will focus on storefront-specific things here.

Your deployment settings are in site settings > build and deploy. They should look like this:

Build command: npm run-script build
Publish directory: dist

It should build successfully, but the site will encounter a couple of problems, discussed below.

Make sure BACKEND_URL is properly set

Update 7 Oct 2019: This has been changed to API_URI.

The storefront prerequisites says: "To run the storefront, you have to set the BACKEND_URL environment to point to the Saleor instance. If you are running Saleor locally with the default settings, set BACKEND_URL to: http://localhost:8000/."

This was not straightforward. Let's recap before continuing: I've deployed saleor to https://example.com on my Ubuntu instance and storefront to https://storefront.example.com on Netlify.

To set the BACKEND_URL, I ran export BACKEND_URL=https://example.com.

Howevever, it returned 404. Looking into console, I found the requested URL to be: https://storefront.example.comhttps://example.com/graphql/

Somehow, it was concatenating the storefront URL to the saleor back end URL.

I thought maybe I had to delete the storefront URL from the graphql query code. Since it's graphql, I thought I had to set the URL in saleor-storefront/apollo.config.js. This was wrong.

After many tries I eventually figured out that I had to open up saleor-storefront/config/webpack/config.base.js and find this section:

new webpack.EnvironmentPlugin({
      "BACKEND_URL": "http://localhost:8000/",
      "SERVICE_WORKER_TIMEOUT": "60000"
    })

I then thought I could simple unset the BACKEND_URL environmental variable and place the correct URL in here. This only reproduced the problem.

The solution is to set the BACKEND_URL environmental variable and completely delete the URL in config.base.js so it's empty:

"BACKEND_URL": "",

Finally, it worked.

Products page won't load, then 404

Once the site was sort of properly up, I tried accessing a product I'd made in the admin dashboard earlier. The page would not load due to a TypeError. Something about price being undefined.

After reloading the page a couple of times, it returned 404 and remained that way.

I tried a whole slew of things I don't remember anymore and finally decided to start from scratch. I ran docker-compose down on the saleor back end, deleted the saleor_web_1 container and saleor_web image that are for the saleor Django back end. I left the redis and celery containers and images alone. I then ran docker-compose up again.

Now it works. But... it loads without the product description.

And, this is where I am right now.

I hope to write as I go along solving issues. Please let me know if you can help with on-going problems.

DEV Community: alex

MongoDB change streams (web SDK) for real time data tracking

Intro

Tech stack

Setup

Set up cluster, database, and/or collection

Create database and/or collection

Give your front end and/or back end client access

Setup Realm App

Create Realm App

Allow user access

Front end

Basic setup

Set up change stream watcher

The filter

Get the party started

Functions, HTTPS endpoints, other features

End

Checklist to estimate time to make a thing

Data

UI/UX (if front end)

Reused code/components

Tests

Meetings

Buffer time

Workflow

Invisible content for screen readers with aria-labelledby

Problem

Solution

Demo

Why not aria-label?

The end

Empty emails from specific address from gmail trash

Problem

Solution

Notes

APIs

Code walkthrough

Some prep

The function itself

Turn on Gmail in Google Services

Set project trigger

Rate limits

Full script

Basic space setup and CRUD with Contentful CLI for development

Basic setup

Get the CLI

Get your CMA

Login on CLI

Create a space

Remember your space

Logout

Set up space with migrations

Delete content types and fields

The end

References

My blog site migration script

How to make URL path prefix for section of website in Gatsby

Usual page template setup in gatsby-node.js

The path prefix

Adapting the blog post template

The end

How to use markdown in pages in Gatsby

Why put markdown in pages?

Basic setup

Create a new project

Install plugins

Make a markdown file

Graphql code

Query response is in props

Method to read markdown

The whole index.js

Starting with saleor-storefront

Set ALLOWED_HOSTS to *

Deploying saleor-storefront to Netlify

Make sure BACKEND_URL is properly set

Products page won't load, then 404

Why not `aria-label`?