DEV Community: Shy Ruparel

Tracking Global Vaccination Rates with Docker, Python, and IoT

Shy Ruparel — Tue, 29 Mar 2022 17:19:09 +0000

I'm a major fan of the ESP32-S2 platform of low-power, low-cost, Wi-Fi capable microcontrollers. Adafruit specifically sells a variant that features a 2.9' E-ink display and Python support, called the MagTag. I'm not ashamed to admit I have far too many of these things littered across my apartment.

One of my favorite organizations is Our World in Data (OWID). They work with thousands of researchers to make research and data accessible to the public. In particular, I've been closely watching their repo that tracks COVID vaccination rates. As numerous worldwide agencies update their vaccination rate metrics, OWID consolidates that data into a single JSON file.

The combination of the MagTag and OWID's data set inspired my first project as the newest member of Docker’s DevRel team. I decided to turn the MagTag into a display for the specific OWID I’m following. However, I ran into a few problems while getting started.

OWID uploads its JSON data collection to GitHub as a raw JSON (or CSV) file, rather than serving it over an API endpoint. The total payload is also 34MB. While that doesn't seem huge, the MagTag itself has just 4MB of flash storage and 2MB of PSRam. That's nowhere near enough memory to parse all that data. Luckily, I commandeered a few spare Raspberry Pi's to create a data processing layer. Using Python, I’m able to spin up a quick Flask server to process the image—and because of Docker, I can deploy a containerized service to my Raspberry Pi in just a few seconds.

My Pi pulls an image built for Armv7 from Docker Hub. That container creates an API endpoint that’s accessible on my local network. From there, the MagTag can request a country's data from the Pi. The Pi grabs the most recent JSON file from OWID, pulls out just the information the MagTag is looking for, and then passes it along. This method reduces the data payload from 34MB down to the KB range, which is much more manageable for an IoT device.

That said, let’s dive into the code.

Coding Your Solution

If you'd rather just read the code, head over to GitHub or Docker Hub for deployment instructions.

Data processing

Knowing that the Raspberry Pi’s data processing function would be pretty lightweight, I decided to spin up a Flask server—taking advantage of Flask's built-in ability to handle JSON with jsonify. The requests library for grabbing our data from OWID was also pretty handy:

# Setup a health route to check if the service is up
@app.route("/")
@app.route("/health")
def hello():
    response = jsonify({"status": "api online"})
    response.status_code = 200
    return response

Secondly, and more importantly, is your route to hit the OWID dataset. This route parses the 34MB JSON file for the requested ISO code, retrieves the most recent data, and formats it for the Raspberry Pi:

# Setup a GET route for requesting the data of the requested ISO Code.
@app.route("/iso_data/<iso>", methods=["GET"])
def iso_data(iso):
    # Get the OWID Data from GitHub
    vaccination_url = "https://raw.githubusercontent.com/owid/covid-19-data/master/public/data/vaccinations/vaccinations.json"
    try:
        vaccinations = requests.get(vaccination_url)
    except:
        return "False"

    vaccinations_dict = {}
    # Using lambda and filter to find our requested ISO Code
    item = list(filter(lambda x: x["iso_code"] == iso, vaccinations.json()))
    if item:
        item = item[0]
        # If data is found format it for our response.
        vaccinations_dict = {
            "data": item["data"][-1],
            "iso_code": item["iso_code"].replace("OWID_", " "),
            "country": item["country"],
        }
        response = jsonify(vaccinations_dict)
        # Returning a 203 since it's a mirror of the OWID data.
        response.status_code = 203
    else:
        # If no data is found return a 404.
        response = jsonify(
            {"status": 404, "error": "not found", "message": "invalid iso code"}
        )
        response.status_code = 404
    return response

Lastly, you should get the code to install the dependencies and generate a requirements file. I use pyenv to manage my Python virtual environments, but you substitute that with your preferred method.

pyenv virtualenv 3.10.1 flask-owid-server
pyenv activate flask-owid-server
pip install flask requests
pip freeze > requirements.txt
flask run

Next, open a new shell prompt and make a curl request to the Flask server, just to confirm that everything is running as expected.

curl localhost:5000
curl localhost:5000/iso_data/USA

Dockerizing and Deploying the Flask Server

With a functional Flask server, we can dockerize it and simplify the app-deployment process for the Raspberry Pi. I started by creating a Dockerfile.

#Use the python 3 base image.
FROM python:3-slim
#Expose ports
EXPOSE 5000
# Setup work directory.
WORKDIR /app
# Copy the requirements file to the container and the install dependencies.
COPY requirements.txt /app
RUN pip3 install -r requirements.txt --no-cache-dir
# Copy all the code into the container/
COPY . /app
# Spin up our flask server.
ENTRYPOINT ["python3"]
CMD ["app.py"]

With this Dockerfile in place, I can run a Docker build command to create my image.

docker build --tag {yourdockerusernamehere}/flask-owid-parser:latest .

You can run your newly-created image from Docker Desktop—while making sure you name the container and set the local port to 5000.

If you prefer using the command line, something like Docker CLI can achieve the same effect.

docker container run --publish 5000:5000 --detach --name flask-owid-parser {yourdockerusernamehere}/flask-owid-parser

Once again, make a curl request to localhost:5000 to confirm that everything is running correctly.

With everything verified on your machine, you can build the application to run on a Raspberry Pi. The Raspberry Pi 3 Model B from this example uses the armv7 platform, so I can create a build specifically targeting armv7. By running this build on my MacBook, the build time will be substantially faster than on my Raspberry Pi.

docker build --platform linux/arm/v7 -t {yourdockerusernamehere}/flask-owid-parser:armv7 .

Back in Docker desktop, you can push your armv7 build to Docker Hub.

Head over to Docker Hub to confirm the armv7 build is displayed correctly.

SSH into your Raspberry Pi to start deploying this code. If you haven't already installed Docker, you can view our documentation for instructions on getting started with Docker on the Pi.

Next, pull the armv7 image from Docker Hub and run it on the Raspberry Pi.

docker pull {yourdockerusernamehere}/flask-owid-parser:armv7
docker run -d -p 5000:5000 --name flask-owid-parser {yourdockerusernamehere}/flask-owid-parser:armv7

Back on my original MacBook, I can both confirm that the server’s running and that it's accessible from my local network—via one final curl request. The default hostname on a fresh Raspberry Pi OS install is raspberrypi, so any Raspberry Pi running Pi OS automatically responds to raspberrypi.local. If that doesn't work for you, swap raspberrypi.local with your Raspberry Pi’s IP address.

ping -c 3 raspberrypi.local
curl raspberrypi.local:5000
curl raspberrypi.local:5000/iso_data/USA

Configuring the MagTag

One of my favorite MagTag features is its Circuit Python support. Forget using pip to install dependencies. Instead, simply copy .mpy onto the MagTag via USB alongside your code. To set up a MagTag for the first time, I'd defer to the excellent tutorials put together by the folks over at Adafruit.

To install this project on your MagTag, visit the GitHub repo (https://github.com/Shy/Docker-MagTag-OWID/tree/main/magtag) containing this project’s code. Drag all code from the MagTag directory onto your MagTag via USB.

If you inspect the code.py file, you'll notice that the MagTag is configured to connect to the Raspberry Pi.

# Build our endpoint
endpoint_iso_first = "http://raspberrypi.local:5000/iso_data/{}".format(
    secrets["iso_first"]
)

Update the secrets.py file to include the information for your own Wifi network. Ensure that your MagTag and Raspberry Pi share the same network.

Reboot the MagTag to see the most recent information—posted by the folks at Our World in Data.

Head over to GitHub to find the full source code for this project with dependencies, and links to the image on Docker Hub. First-time Python and Docker users should browse our quick start tutorial for Python devs.

App hosting with the Contentful App Framework

Shy Ruparel — Wed, 23 Jun 2021 14:52:04 +0000

I'm cross-posting this article for my Colleague David Fateh. Check out the original post over on the Contentful Blog.

When you develop an app for Contentful using the App Framework, you create a single-page application that runs inside an iframe and shows up in the Contentful UI. Previously, hosting an app on the App Framework was only possible through third-party solutions. Today, we are announcing an easier method to host your app called app hosting!

App hosting takes the hassle out of finding a hosting provider for your app. We have provided developers with a tool to upload their application in bundles which provides for many great development features.

An app bundle is your single-page application directory. Each bundle should contain at least an index.html file, which will serve as the main entry point for your application to load. A typical bundle's file tree will look something like this:

build
├── index.html
└── static
    ├── css
    │   └── index.css
    └── js
        └── index.js

Where build is your root directory and, inside it, there is an index.html file, along with any assets such as CSS and JavaScript that would get loaded alongside your index.html file.

While there are many ways to create apps, we provide a quickstart tool, the create-contentful-app, which can get you a jump-start on your app. If you aren't familiar with building apps, I'd suggest checking out our livestream where Felix and I build the Contentful starter tutorial app.

Now, let's take a look at the three easy steps to bundling and hosting your own app.

Step 1

If you built your app using the create-contentful-app or create-react-app, you can simply run

npm run build

If you built your app using another tool or from scratch, your directory structure will need a minimum of an index.html file at the root.

Again, a simple app directory structure would look something like this:

build
├── index.html
└── static
    ├── css
    │   └── index.css
    └── js
        └── index.js

Step 2

Navigate to your app's AppDefinition found in your organization settings.

Step 3

Toggle the app hosting option found under "Frontend" and drag and drop your build directory into the highlighted area like so:

That's it! Your app is now hosted via Contentful. In the Bundles tab, you can check out multiple build versions and revert to an older version of your app in case you need to roll back changes.

While uploading an app is easy, there are some common errors that could occur:

Your app doesn't have an index.html file in the root of the uploaded bundle.
You are uploading a Zip that encapsulates the root directory instead of the content directly. Most of the time, the common fix is to navigate into your root directory, select all your files, and zip it at that level.
Your upload succeeds but the app doesn't load or appear in the UI. This could be caused by an error in your app's index.html or the JavaScript that it executes. A look at the developer console should provide some hints.

To learn more about the technical details of app hosting, head over to our documentation.

Launching app hosting on Contentful is another step forward in our vision of creating an extensible and cohesive set of tools that connect our projects and our content. If you want to chat about what you're working on, feel free to ping me on our community Slack where I work with fellow developers as we take steps to make our App Framework more accessible and versatile for everyone.

How to build a multi-location app for the Open Graph protocol

Shy Ruparel — Wed, 23 Jun 2021 14:44:09 +0000

I'm cross-posting this article for my Colleague David Fateh. Check out the original post over on the Contentful Blog.

If you are a content creator for the web, you may have used the Open Graph protocol. The Open Graph protocol is a way to display and share information about your resource on the internet. You may have seen these in action when sharing websites and URLs on Twitter and Facebook, or when you are chatting with your coworkers on Slack. The preview and metadata for the link is shown to the user as in the Twitter post below.

Building an app on the App Framework can be as simple as creating a single custom button or as complicated as composing multiple components in different locations of Contentful. I want to run through an example app I've created which makes use of the App Framework's ability to handle complexity, but more importantly shows how to pass data between different app locations.

In Contentful, we can create our own Open Graph data by making use of a simple content type that provides the user with the necessary fields to show these kinds of previews. Then, using the App Framework, we can augment the editor experience to allow for automated title generation of our content as well as a visual preview.

In this post, we will cover a two things: 1. Creating a content type to handle simple Open Graph data. 2. Creating an app which helps visualize the data and allows for a preview mode. 3. When all is said and done, our Open Graph content type will still need to be pulled from a Contentful API response when we generate our web pages. The information contained in an Open Graph entry will be enough information to populate meta tags in the head of our HTML filles. Let's get started!

Creating the content type

We can start by creating a new content type in our space called "Open Graph," which will have five fields:

Title: short text
Content: single reference (in our case, referencing blog posts)
Type: short text
Image: media
URL: short text

This content type does a few things. First, it allows editors to pick a reference entry. In many cases, this could be a blog post, web page, link or other type of media. In our example, we will be using the Open Graph content type in a blog. This means our Content reference field will reference "Blog Post" content types.

Second, we can automate the "Title" field by asking the editor to select a blog post and grabbing the title from that post. This is something we will look at when we dive into the code.

Lastly, we can create a custom button that will show the editor a preview of our Open Graph data as it might appear in a card UI such as Twitter or Facebook.

Now that we have our content type set up, let's start developing our app.

Creating the app

For those that are familiar with the App Framework and want to see the source code of the app, you can see it here on GitHub. For those less familiar, I've written a few tutorials in the past about how to build apps. We also have a tutorial for beginners, Building your first app, where we explain all the necessary steps to get up and running.

An app is a single page application that runs inside of an iframe and can be rendered in different locations inside of the Contentful Web App. While building an app can be done using any framework (or lack thereof), I'm going to use the Create Contentful App CLI tool, which will bootstrap a React app with all the necessary logic required to get up and running inside of Contentful.

To initialize and run our app locally, run these commands in your terminal:

npx @contentful/create-contentful-app init open-graph
cd open-graph
npm start

The app is now running as a single page application on http://localhost:3000. In order to see our app in Contentful, we will need to create an AppDefinition and assign our app to show up in the correct locations.

Creating the AppDefinition

An AppDefinition is the entity that represents an app in Contentful. It contains general app information, like where it is visible. It also provides settings that enable it to run independently, or enable particular settings for any current and future installations.

Note: In order to create an AppDefinition, you must be a Contentful organization admin or developer. If you are not, you can ask your admin for access or sign up for a free Contentful account to use as a sandbox.

Let's create a new AppDefinition. Go to your organization settings. In the top menu, select Apps. From here we will create a new AppDefinition with the following properties:

Name: Open Graph
App URL: http://localhost:3000 (remember this is running on our machine)
First location: Field - short text
Second location: Page

Once filled in, hit the Create button in the top-right corner. Next, let's install our app into the space where we created our Open Graph content type.

Installing and developing the app

To install the app, head over to the space where we created our Open Graph content type. In the Apps menu bar, click "Manage apps" then find the Open Graph app in the list of available apps (it will show up with a private tag next to it). Click Install.

Once our app is installed, it is ready to be assigned to our Open Graph content type. Navigate to the Content Model page and select the Open Graph content type. In there we will modify the settings of our "Title" field by navigating to the appearance area selecting our Open Graph app. Click save.

Now, create a new Open Graph entry to see how the app renders. We can see that our app is rendering by the message that is showing up in our Title field. Of course, we'd like it if our title field actually showed a title instead of a developer message, so at this point it is time to dive into the code that was created earlier by our create-contentful-app CLI tool.

To start, create a field that shows a title based off of the reference blog post in our Content field. To get this functionality, we will make use of Forma 36 --- the Contentful Design System to implement UI elements which have the same look and feel as Contentful itself. We're going to use a TextInput which will be disabled for manual editing but will automatically pull the title information from our content reference field.

In our code, modify src/components/Field.tsx to create a better experience.

To start, import a few libraries that come out of the box when we ran the create-contentful-app CLI commands:

import React, {useState, useEffect} from 'react';
import tokens from '@contentful/forma-36-tokens'
import { TextInput, Note, Button } from '@contentful/forma-36-react-components';
import { FieldExtensionSDK } from '@contentful/app-sdk';

We are using React here for our UI. Tokens is a Forma 36 utility that will provide us with certain variables like color and sizing. The Forma 36 components also come out of the box and will be useful for constructing our UI. Since we are using TypeScript, I'm also including the SDK typings by default.

Next, we'll want to modify the component itself to handle state, pull information from another field and render some UI.

First, create some state:

const Field = (props: FieldProps) => {
  // Get the title of the field and create a setter
  const [title, setTitle] = useState(props.sdk.field.getValue() || null);

Next, listen for changes to our content reference field. When the content field has been populated with data, grab the entry, find the title and set that as the title in state. Also save that as the title field's value in the Open Graph content type.

  useEffect(() => {
    // Resize the field so it isn't cut off in the UI
    props.sdk.window.startAutoResizer();

    // when the value of the `content` field changes, set the title
    props.sdk.entry.fields.content.onValueChanged((entry) => {
      if (!entry) {
        setTitle(null);
        return;
      }

      // The content field is a reference to a blog post.
      // We want to grab that full entry to get the `title` of it
      props.sdk.space.getEntry<BlogPost>(entry.sys.id).then((data) => {
        if (data.fields.title['en-US'] !== title) {
          const title = data.fields.title['en-US'];
          setTitle(title);
          props.sdk.field.setValue(title);
        }
      });
    });
  });

Last, we can render our UI based on the above logic to give editors some validation hints. We can either warn that a reference needs to be picked, or if a reference is picked, to use the correct title information in our field. We are also going to include a button, which will open up our page location by passing the entry's ID. We will touch on that more below.

  if (title === null) {
    return <Note noteType="warning">Link a blog post below to assign a title.</Note>
  }

  return (
    <>
      <TextInput disabled value={title} style={{marginBottom: tokens.spacingM}}/>
      <Button onClick={() => props.sdk.navigator.openCurrentAppPage({path: `/${props.sdk.entry.getSys().id}`})} buttonType="naked">Preview Open Graph</Button>
    </>
  );
};

Our full component code looks like this now:

import React, {useState, useEffect} from 'react';
import tokens from '@contentful/forma-36-tokens'
import { TextInput, Note, Button } from '@contentful/forma-36-react-components';
import { FieldExtensionSDK } from '@contentful/app-sdk';

interface FieldProps {
  sdk: FieldExtensionSDK;
}
// Custom type to denote a blog post content type
interface BlogPost {
  sys: {
    id: string;
  };
  fields: {
    body: object;
    title: {
      'en-US': string;
    };
  };
};

const Field = (props: FieldProps) => {
  // Get the title of the field and create a setter
  const [title, setTitle] = useState(props.sdk.field.getValue() || null);

  useEffect(() => {
    // Resize the field so it isn't cut off in the UI
    props.sdk.window.startAutoResizer();

    // when the value of the `content` field changes, set the title
    props.sdk.entry.fields.content.onValueChanged((entry) => {
      if (!entry) {
        setTitle(null);
        return;
      }

      // The content field is a reference to a blog post.
      // We want to grab that full entry to get the `title` of it
      props.sdk.space.getEntry<BlogPost>(entry.sys.id).then((data) => {
        if (data.fields.title['en-US'] !== title) {
          const title = data.fields.title['en-US'];
          setTitle(title);
          props.sdk.field.setValue(title);
        }
      });
    });
  });

  if (title === null) {
    return <Note noteType="warning">Link a blog post below to assign a title.</Note>
  }

  return (
    <>
      <TextInput disabled value={title} style={{marginBottom: tokens.spacingM}}/>
      <Button onClick={() => props.sdk.navigator.openCurrentAppPage({path: `/${props.sdk.entry.getSys().id}`})} buttonType="naked">Preview Open Graph</Button>
    </>
  );
};

export default Field;

Now that we have a field that can automatically pull a title from the referenced blog post, we also want to build out a page location, which will show a preview of the Open Graph to editors when they click the preview button. The button is under our Title and says "Preview Open Graph." This button will direct us to our page location which we will create soon.

Let's start by modifying our src/components/Page.tsx file. Again, we are going to import a few things from Forma 36 in order to build a nice preview UI for our Open Graph components.

import React, { useState, useEffect } from 'react';
import {
    Card,
    Note,
    Heading,
    Paragraph,
    Typography,
    TextLink,
    Button,
    SkeletonBodyText,
    SkeletonContainer,
    SkeletonDisplayText,
} from '@contentful/forma-36-react-components';
import tokens from '@contentful/forma-36-tokens';
import { PageExtensionSDK } from '@contentful/app-sdk';

In the Page component, check the URL parameters that we passed in when the editor clicked the button:

const Page = (props: PageProps) => {
    // false indicates an error occurred
    // otherwise the entry is loading or loaded
    const [entry, setEntry] = useState<OpenGraphPreview | false>();

    useEffect(() => {
        // Get the entry ID which is passed in via the URL
        const entryId = props.sdk.parameters?.invocation?.path.replace('/', '');

        // If no entry ID exists, show an error message by setting content to false
        if (!entryId) {
            setEntry(false);
            return;
        }

We use sdk.parameters.invocation.path to access the custom path in the URL of our app, which was passed in by sdk.navigator.openCurrentAppPage from our field component as the on click function of the button.

Now, complete the useEffect function with some network calls to get all the data we need to display the preview:

        // Get the entry data by getting the linked asset and content body
        props.sdk.space.getEntry<OpenGraphEntry>(entryId).then((data) => {
            Promise.all([
                // Grabs the Image asset of the Open Graph content type
                props.sdk.space.getAsset<Asset>(
                    data.fields.image['en-US'].sys.id
                ),
                // Grabs the long text body of the blog post
                props.sdk.space.getEntry<Content>(
                    data.fields.content['en-US'].sys.id
                ),
            ]).then(([asset, content]) => {
                // combine the data from the two `space` calls
                setEntry({
                    title: data.fields.title['en-US'],
                    imageUrl: asset.fields.file['en-US'].url,
                    previewBody: content.fields.body['en-US'],
                    url: data.fields.url['en-US'],
                    id: entryId,
                });
            });
        });
    }, []);

Notice that we're using Promise.all to get data about the entry and the image asset which we will use in our preview.

Now render the component UI:

    if (entry === false) {
        return <Note noteType="negative">Error retrieving entry!</Note>;
    }

    return (
        <div
            style={{
                display: 'flex',
                width: '100%',
                height: '100vh',
                alignItems: 'center',
                marginTop: tokens.spacingXl,
                flexDirection: 'column',
            }}
        >
            <div style={{ justifyContent: 'center' }}>
                <SkeletonContainer>
                    <SkeletonDisplayText numberOfLines={1} />
                    <SkeletonBodyText numberOfLines={3} offsetTop={35} />
                </SkeletonContainer>
            </div>
            <Typography>
                <Heading>
                    Open Graph Preview
                </Heading>
            </Typography>
            <Card style={{ width: '260px' }}>
                {!entry ?
                <SkeletonContainer>
                    <SkeletonDisplayText numberOfLines={1} />
                    <SkeletonBodyText numberOfLines={3} offsetTop={35} />
                </SkeletonContainer> :
                <>
                    <Typography>
                        <TextLink href={entry.url}>{entry.title}</TextLink>
                        <Paragraph>{entry.previewBody}</Paragraph>
                    </Typography>
                    <img src={entry.imageUrl} alt="" style={{ width: '200px' }} />
                </>}
            </Card>
            <Button
                buttonType="muted"
                disabled={!entry}
                onClick={() => props.sdk.navigator.openEntry(entry!.id)}
                style={{ margin: `${tokens.spacingXl} 0` }}
            >
                Back to entry
            </Button>
            <div style={{ justifyContent: 'center' }}>
                <SkeletonContainer>
                    <SkeletonDisplayText numberOfLines={1} />
                    <SkeletonBodyText numberOfLines={10} offsetTop={35} />
                </SkeletonContainer>
            </div>
        </div>
    );
};

For the most part, I am using Skeleton components, which are for looks; they show a loading preview. The Card component is used for actually displaying the content in the Open Graph card. All together, the Page component code looks like this:

import React, { useState, useEffect } from 'react';
import {
    Card,
    Note,
    Heading,
    Paragraph,
    Typography,
    TextLink,
    Button,
    SkeletonBodyText,
    SkeletonContainer,
    SkeletonDisplayText,
} from '@contentful/forma-36-react-components';
import tokens from '@contentful/forma-36-tokens';
import { PageExtensionSDK } from '@contentful/app-sdk';

interface PageProps {
    sdk: PageExtensionSDK;
}

// An open graph content type
interface OpenGraphEntry {
    fields: {
        title: {
            'en-US': string;
        };
        type: {
            'en-US': string;
        };
        url: {
            'en-US': string;
        };
        image: {
            'en-US': {
                sys: {
                    id: string;
                };
            };
        };
        content: {
            'en-US': {
                sys: {
                    id: string;
                };
            };
        };
    };
    sys: {
        id: string;
    };
}

// A custom shape for the entry we are going to render in our UI
interface OpenGraphPreview {
    id: string;
    title: string;
    imageUrl: string;
    previewBody: string;
    url: string;
}

interface Content {
    fields: {
        body: {
            'en-US': string;
        };
    };
}

interface Asset {
    fields: {
        file: {
            'en-US': {
                url: string;
            };
        };
    };
}

const Page = (props: PageProps) => {
    // false indicates an error occurred
    // otherwise the entry is loading or loaded
    const [entry, setEntry] = useState<OpenGraphPreview | false>();

    useEffect(() => {
        // Get the entry ID which is passed in via the URL
        // @ts-ignore
        const entryId = props.sdk.parameters?.invocation?.path.replace('/', '');

        // If no entry ID exists, show an error message by setting content to false
        if (!entryId) {
            setEntry(false);
            return;
        }

        // Get the entry data by getting the linked asset and content body
        props.sdk.space.getEntry<OpenGraphEntry>(entryId).then((data) => {
            Promise.all([
                // Grabs the Image asset of the Open Graph content type
                props.sdk.space.getAsset<Asset>(
                    data.fields.image['en-US'].sys.id
                ),
                // Grabs the long text body of the blog post
                props.sdk.space.getEntry<Content>(
                    data.fields.content['en-US'].sys.id
                ),
            ]).then(([asset, content]) => {
                // combine the data from the two `space` calls
                setEntry({
                    title: data.fields.title['en-US'],
                    imageUrl: asset.fields.file['en-US'].url,
                    previewBody: content.fields.body['en-US'],
                    url: data.fields.url['en-US'],
                    id: entryId,
                });
            });
        });
    }, []);

    if (entry === false) {
        return <Note noteType="negative">Error retrieving entry!</Note>;
    }

    return (
        <div
            style={{
                display: 'flex',
                width: '100%',
                height: '100vh',
                alignItems: 'center',
                marginTop: tokens.spacingXl,
                flexDirection: 'column',
            }}
        >
            <div style={{ justifyContent: 'center' }}>
                <SkeletonContainer>
                    <SkeletonDisplayText numberOfLines={1} />
                    <SkeletonBodyText numberOfLines={3} offsetTop={35} />
                </SkeletonContainer>
            </div>
            <Typography>
                <Heading>
                    Open Graph Preview
                </Heading>
            </Typography>
            <Card style={{ width: '260px' }}>
                {!entry ?
                <SkeletonContainer>
                    <SkeletonDisplayText numberOfLines={1} />
                    <SkeletonBodyText numberOfLines={3} offsetTop={35} />
                </SkeletonContainer> :
                <>
                    <Typography>
                        <TextLink href={entry.url}>{entry.title}</TextLink>
                        <Paragraph>{entry.previewBody}</Paragraph>
                    </Typography>
                    <img src={entry.imageUrl} alt="" style={{ width: '200px' }} />
                </>}
            </Card>
            <Button
                buttonType="muted"
                disabled={!entry}
                onClick={() => props.sdk.navigator.openEntry(entry!.id)}
                style={{ margin: `${tokens.spacingXl} 0` }}
            >
                Back to entry
            </Button>
            <div style={{ justifyContent: 'center' }}>
                <SkeletonContainer>
                    <SkeletonDisplayText numberOfLines={1} />
                    <SkeletonBodyText numberOfLines={10} offsetTop={35} />
                </SkeletonContainer>
            </div>
        </div>
    );
};

export default Page;

In the final code, I've added some Typescript interfaces to ensure we are working with the correct data structures. If you'd like to check out the complete code, visit the repo: davidfateh/ctfl-open-graph: Multi location app for Open Graph.

If you'd like to go beyond this functionality, Salma recently wrote a post on solving this issue using Puppeteer and Node.js to generate this data programmatically.

Final thoughts

Passing data between app locations can be very useful for creating cohesive experiences for editors as they work on content that is meant to be visualized in different ways. The Open Graph app is one example of how retrieving entry data and passing it around inside of an app can extend some basic functionality.

If you have an app idea or are working on a prototype, our Slack Community can be very helpful if you want to share or get help implementing ideas. You can take advantage of some of the cool things we are doing and discussing over there. Our extensibility team and I are active in our community, and we are always curious to hear ideas about your next big projects!

App Framework tutorial: Building a custom reference app

Shy Ruparel — Wed, 23 Jun 2021 14:40:52 +0000

I'm cross-posting this article for my Colleague David Fateh. Check out the original post over on the Contentful Blog.

Shy and I set out on a livestream to customize the Contentful reference field by using the App Framework to build a custom React app that would provide us with some new and novel functionality. While our progress during the live stream was good, I took the liberty of tightening up the styling and making sure additional functionality was possible. In this post, I'm going to show you how I created a custom reference field app which provides custom functionality.

Building on the App Framework

Building an app on the App Framework as a developer requires an understanding of how editors --- the people who use the Contentful web app --- may want to manipulate and view entry data. We know that everyone has different use cases and so the customization allowed by the App Framework can be useful when trying to create an experience that will help solve these unique cases.

When you define fields in an entry, you are defining how data should be displayed. In the case of the reference field, data such as the title of the entry is displayed by default like in the screenshot below:

We can enhance this experience by creating a visually similar custom app using the App Framework that can show more data from the referenced entry such as the body of the post, not just the title.

Getting started

To start, we used the create-contentful-app CLI tool. The tool is available for free to developers, and they can use it to quickly get an app up and running. The create-contentful-app CLI tool creates a React app project with Forma 36 (our design library) and our open source field editors for easy access.

We get all this by running the commands below:

npx @contentful/create-contentful-app init reference-field-app
cd reference-field-app
npm start

At this point our app is running on localhost:3000 but won't be accessible until we create the AppDefinition and select the locations where we want it to show up. Let's do this next.

Creating the AppDefinition

An AppDefinition is the entity that represents an app in Contentful. You can think of it as a sort of blueprint for how your app will interact inside the Contentful experience.

You must have an admin or developer account in your Contentful organization. Many developers find it easiest to create a free Contentful organization for a developer environment. You can also develop an app in your primary organization if you prefer to develop in a space or environment which isn't production facing.

To create the AppDefinition, head to your organization settings and click on Apps in the top menu bar. Once on the AppDefinition page, click the button to create a new app.

First, we are going to name our app: Custom Reference Field. Next, we will make its app URL http://localhost:3000. This is where our local app is currently running. Last, we are going to select the field location and pick the reference field (many) and click the confirm button to save this AppDefinition.

Replacing a built-in reference field with our custom app

Let's now head over to a space where we want to see our app show up. In the top menu of our space or environment, click Apps then Manage apps. From here we can find our newly created Custom Reference Field app and install it into our space.

Since we are building an app that will take over a reference field, we must also have a content type that makes use of a reference field. For our purposes, we are going to use a blog post list content type as an example.

In our content model, we are going to adjust the field of our blog post list to use our app by editing the settings of the reference field and selecting appearance. Let's choose our newly created Custom Reference Field app.

Coding the app for custom functionality

Once the app is assigned to a field, we can head over to our entries section and find an entry to see how the app is displayed. In our example, we have a reference field which links to different blog posts. Each blog post has a rich text field which we'd like to use to display some more custom information --- something that will give us a bit more functionality than the default reference field experience.

Let's create some custom functionality. First, we are going to replace our Hello World component with components from Forma 36, as well as utilities from React and a Contentful rich text renderer for display use. We are also going to use the handy MultipleEntryReferenceEditor which is part of our open source editor package. We are using the rich text renderer in this situation because our referenced content model of blog posts uses a rich text field for its body. Depending on the content type, you are referencing in your custom reference field app, displaying this data through different custom or open source components is possible.

import React, { useEffect } from 'react';
import {
    Card,
    Typography,
    Heading,
    CardActions,
    DropdownList,
    DropdownListItem,
} from '@contentful/forma-36-react-components';
import { MultipleEntryReferenceEditor } from '@contentful/field-editor-reference';
import { documentToReactComponents } from '@contentful/rich-text-react-renderer';
import { FieldExtensionSDK } from '@contentful/app-sdk';

Next, let's replace the Field component with custom code using the MultipleEntryReferenceEditor. You'll notice that while this markup is not very complicated, we will have to implement a customRenderer for our MultipleEntryReferenceEditor to show properly.

const Field = (props: FieldProps) => {
    useEffect(() => {
        props.sdk.window.startAutoResizer();
    });

    return (
        <MultipleEntryReferenceEditor
            renderCustomCard={customRenderer}
            viewType="link"
            sdk={props.sdk}
            isInitiallyDisabled
            hasCardEditActions
            parameters={{
                instance: {
                    showCreateEntityAction: true,
                    showLinkEntityAction: true,
                },
            }}
        />
    );
};

Let's create the custom renderer function which will render some more React components:

 const customRenderer = (props: any) => {
    if (props.contentType.sys.id !== 'blogPost') {
        return false;
    }
    const title = props.entity.fields?.title?.[props.localeCode] || 'Untitled';

    return (
        <Card style={{ flexGrow: 1 }} padding="none">
            <div style={{ display: 'flex' }}>
                <div>{props.cardDragHandle}</div>
                <div style={{ flexGrow: 1, padding: '1em' }}>
                    <Typography style={{ marginBottom: '20px' }}>
                        <Heading style={{ borderBottom: '1px solid gray' }}>
                            {title}
                        </Heading>
                        {props.entity.fields.body &&
                            documentToReactComponents(
                                props.entity.fields.body[props.localeCode]
                            )}
                    </Typography>
                </div>
                <div style={{ padding: '1em' }}>
                    <CardActions>
                        <DropdownList>
                            <DropdownListItem onClick={props.onEdit}>
                                Edit
                            </DropdownListItem>
                            <DropdownListItem onClick={props.onRemove}>
                                Remove
                            </DropdownListItem>
                        </DropdownList>
                    </CardActions>
                </div>
            </div>
        </Card>
    );
};

Now let's put it all together:

import React, { useEffect } from 'react';
import {
    Card,
    Typography,
    Heading,
    CardActions,
    DropdownList,
    DropdownListItem,
} from '@contentful/forma-36-react-components';
import { MultipleEntryReferenceEditor } from '@contentful/field-editor-reference';
import { documentToReactComponents } from '@contentful/rich-text-react-renderer';
import { FieldExtensionSDK } from '@contentful/app-sdk';

interface FieldProps {
    sdk: FieldExtensionSDK;
}

const customRenderer = (props: any) => {
    if (props.contentType.sys.id !== 'blogPost') {
        return false;
    }
    const title = props.entity.fields?.title?.[props.localeCode] || 'Untitled';

    return (
        <Card style={{ flexGrow: 1 }} padding="none">
            <div style={{ display: 'flex' }}>
                <div>{props.cardDragHandle}</div>
                <div style={{ flexGrow: 1, padding: '1em' }}>
                    <Typography style={{ marginBottom: '20px' }}>
                        <Heading style={{ borderBottom: '1px solid gray' }}>
                            {title}
                        </Heading>
                        {props.entity.fields.body &&
                            documentToReactComponents(
                                props.entity.fields.body[props.localeCode]
                            )}
                    </Typography>
                </div>
                <div style={{ padding: '1em' }}>
                    <CardActions>
                        <DropdownList>
                            <DropdownListItem onClick={props.onEdit}>
                                Edit
                            </DropdownListItem>
                            <DropdownListItem onClick={props.onRemove}>
                                Remove
                            </DropdownListItem>
                        </DropdownList>
                    </CardActions>
                </div>
            </div>
        </Card>
    );
};

const Field = (props: FieldProps) => {
    useEffect(() => {
        props.sdk.window.startAutoResizer();
    });

    return (
        <MultipleEntryReferenceEditor
            renderCustomCard={customRenderer}
            viewType="link"
            sdk={props.sdk}
            isInitiallyDisabled
            hasCardEditActions
            parameters={{
                instance: {
                    showCreateEntityAction: true,
                    showLinkEntityAction: true,
                },
            }}
        />
    );
};

export default Field;

We now have a working component that not only shows the title of the blog post but also the first line of the blog post. We have successfully transformed the default experience of the reference field to show data that is more conducive for our use-case. If you're interested in seeing the full code, check out the repo here.

Wrapping up

For our purposes, changing the fields inside of an entry can be a very powerful tool. There are many cases for why you may want to either slightly modify existing functionality or totally recreate your own. For the cases where you'd like to simply modify the web app, we provide the default fields as a React component in our open source editor library. For cases where you'd like to create your own experience, Forma 36 can be an invaluable tool for achieving a very fluid look and feel of your UI without having to spend time messing with layout yourself.

That said, it is always interesting to see the different ways developers have come together and built their own components for the UI/UX they envision for their users. Many developers make use of our Slack Community where help and ideas are easily shared. I'm active on the channel and am always happy to help explore ideas or guide other developers through the app creation process.

If you'd like to join our community, you can take advantage of some of the cool things we are doing and discussing over there. If you are interested in seeing more video tutorials, check out our weekly streams on Twitch and YouTube where we code live and work through problems on the fly!

If you haven't signed up for a free Contentful account yet, register for a Community edition!

Visualize your Content Models with ContentModel.io

Shy Ruparel — Wed, 26 May 2021 14:15:37 +0000

Developer Advocate Shy Ruparel takes a look at ContentModel.io, a tool to visualize Content Models. ContentModel.io also let's you share your own content models and import the models that others have made.

How to post to Twitter using Contentful webhooks and IFTTT

Shy Ruparel — Mon, 24 May 2021 14:30:03 +0000

Want to post to Twitter every time you publish a blog post? Don't have the time to build a Twitter API integration? Contentful Developer Evangelist Shy Ruparel shows us how easy it is to set up social media posting using Contentful webhooks and IFTTT (If This Then That).

Contentful Updates: Introducing new APIs for managing bulk content

Shy Ruparel — Wed, 12 May 2021 14:59:25 +0000

Every month Shy from the Contentful Dev Rel Team will share his favorite changes from the Contentful Changelog.

This month is so jam packed he's made two videos. For this one he's sharing all of the details about the three newest Contentful APIs for managing content in Bulk Content

How does GraphiQL work? An intro to Introspection.

Shy Ruparel — Wed, 12 May 2021 14:36:02 +0000

Contentful Developer Evangelist Shy Ruparel took a deep dive into figuring out how does GraphiQL build the self documentation that we're all come to love as a major feature of the tool.

Contentful Updates: Tags and Tag Based Permissions

Shy Ruparel — Wed, 12 May 2021 14:27:55 +0000

Every month Shy from the Contentful Dev Rel Team will share his favorite changes from the Contentful Changelog.

This month is so jam packed he's made two videos. For this one he's sharing all of the details about Tags and Tag based Permissions.

Tags are user-defined keywords, consisting of an editable label and a permanent ID, that can be used to create arbitrary content collections within a given environment.

The workflow for using tags requires a space admin (or other authorized roles) to create a list of tags within the space. After that, users can apply tags to individual entries/assets or bulk tag multiple entries/assets.

Contentful Changes: Sys Comparisons, Webhooks and App updates

Shy Ruparel — Fri, 19 Mar 2021 15:39:25 +0000

Every month Shy from the Contentful Dev Rel Team will share his favorite changes from the Contentful Changelog.

This month he's sharing number and date comparison filters for "sys" properties in GraphQL, updates to Webhooks and the App Framework.

Showcasing the Contentful blog with E-Ink, CircuitPython and IoT

Shy Ruparel — Thu, 18 Mar 2021 18:13:28 +0000

In 2007, the New York Times curated an art installation by Mark Hansen and Ben Rubin called “Moveable Type.” Mark and Ben had taken headlines and quotes from the newspaper’s more than 150-year-old archive and rendered them onto 560 small screens across the lobby of the building. After seeing the exhibit for the first time in 2014, I’ve made a point to visit every time I pass through Times Square.

Photo by Ben Rubin

I had a chance to hear a collaborator, Marty Chafkin from Perfection Electricks, talk about the project at Brooklyn Research back in 2019. He mentioned the data collection process and how once they’d pulled the data out of the NYT archive and placed it into each device, there wasn’t an easy way to update each screen. I also noticed that it would be difficult to find the articles referenced on the screens.

At the time, I thought, “Wow, Contentful would be a great use case for this.” But not having access to any low-power screens at the time, I assumed this was a project I wouldn’t have a chance to work on anytime soon.

Late last year, Adafruit, an open-source hardware company based in New York City that creates products to help folks learn electronics, released the Adafruit MagTag. The MagTag is a 2.9-inch E-ink display combined with an ESP32-S2 wireless module. It’s a low-power IoT display that you can code using Python. It has four built-in colorful miniature LEDs alongside four buttons.

I ordered one as soon as I could, deciding that this would be the opportunity to revisit making my version of Moveable Type.

The MagTag buttons would make it possible to send the user a notification that included a link to whatever content the screen was displaying. I reached out to Aydrian Howard from Courier to help me with this final part. Aydrian and I did some live coding together back in November of 2020, and he shared how you could use the Courier API to design and send notifications. He recommended sharing the content via Discord and, luckily, he was free to build the project with me live on stream.

Let’s look at the code

If you’d rather see the code in full, head over to GitHub.

One of the reasons that I like the MagTag so much is that it supports CircuitPython. CircuitPython is a version of Python that adds hardware support to the standard Python implementation. Unlike a traditional Python project, you install dependencies by dragging .mpy files onto the board of the Microcontroller rather than using a package manager like pip. Since most IoT circuit boards don’t come with a ton of memory anyway — and I didn’t want to compile the Contentful Python SDK into an .mpy file — I chose to reach out to the Contentful API via HTTP.

Adafruit provides some precompiled .mpy files of the most useful libraries for CircuitPython. We’ll be using the Adafruit version of requests since it’s precompiled into an .mpy file. To get started, import all of the required dependencies, grab your environment variables and connect to the WiFi.

import time
import gc
import wifi
import random
import adafruit_requests
import ssl
import socketpool
import terminalio
from adafruit_magtag.magtag import MagTag

magtag = MagTag()

# Add a secrets.py to your filesystem that has a dictionary called secrets with "ssid" and
# "password" keys with your WiFi credentials. DO NOT share that file or commit it into Git or other
# source control.
# pylint: disable=no-name-in-module,wrong-import-order
try:
    from secrets import secrets
except ImportError:
    print("Credentials and tokens are kept in secrets.py, please add them there!")
    raise

print("Connecting to %s" % secrets["ssid"])
wifi.radio.connect(secrets["ssid"], secrets["password"])
print("Connected to %s!" % secrets["ssid"])

pool = socketpool.SocketPool(wifi.radio)
requests = adafruit_requests.Session(pool, ssl.create_default_context())

Luckily my last post was all about using GraphQL via HTTP. I reused my previous Python code and made a few changes to account for the different queries and the API keys of the production Contentful blog. Using our GraphQL API rather than the traditional Contentful Rest API came with the benefit of reducing the number of calls I would need to make to the Contentful API, which reduced the payload of the response.

Fetching content

To get started, make a call to grab the last 100 blog posts from contentful.com/blog.

endpoint = "https://graphql.contentful.com/content/v1/spaces/%s/" % (
    secrets["space_id"]
)
headers = {"Authorization": "Bearer %s" % (secrets["CDA_token"])}

# Get Dev Blog Posts
query = """query {
  blogPostCollection {
    items {
      sys {
        id
      }
    }
  }
}
"""

print("Making blog post collection query.")
response = requests.post(endpoint, json={"query": query}, headers=headers)

Next, pick one of those blog posts at random and grab all of the information we need to display the article on the E-ink screen. Since memory is a constraint, overwrite the previously used queries and requests rather than setting a new variable.

# Get Individual Post
query = """{
  blogPost(id: \"%s\") {
    title
    publishDate
    slug
    authorsCollection {
      items {
        name
      }
    }
    introduction
  }
}
""" % (
    random.choice(response.json()["data"]["blogPostCollection"]["items"])["sys"]["id"]
)

print("Making blog post query.")
response = requests.post(endpoint, json={"query": query}, headers=headers)

With that request, we’re able to parse everything into a human-readable format and display it on the screen. Since the screen itself is only 2.9 inches, we’ll only use the first 170 characters of each blog post.

The MagTag comes with functions to create and set text boxes. When the box is ready, we’re able to set the text and let the screen know if it needs to refresh. Since we’re editing four text boxes for each item we’ll be displaying on screen, we can hold off triggering the refresh event until the final text item is set.

# Formatting for the author text
magtag.add_text(
    text_font="/fonts/Arial-Bold-12.pcf",
    text_position=(10, 38),
)

author_string = ""
for author in response.json()["data"]["blogPost"]["authorsCollection"]["items"]:
    if author_string == "":
        author_string = author["name"]
    else:
        author_string = author_string + " & " + author["name"]

magtag.set_text(
    val=author_string,
    index=1,
    auto_refresh=False,
)

In this snippet, I created a text box for the author field. Next I generated a single string for our authors, looping over the author array as it’s possible a blog post has multiple authors. Lastly, I set the text box with the generated string. Since this is only the second of four text boxes we’ll need to change, we tell the display not to update. If you want to see the full display code, head over to GitHub.

Sending information to the Courier API

I wanted to send the user a link to the full article when they hit one of the MagTag buttons. Aydrian proved essential for achieving this. He let me know that we could take advantage of the Courier API’s ability to send notifications via a single HTTP request.

Just like before, we overwrite the previously used variables to make a request. Since we’ll be sharing information from the article, include that information in the request body. We’ll be able to use that information from Courier to design a notification later on. Lastly, make sure to set an event. Courier will use the information to figure out what should specifically happen to that specific information.

endpoint = "https://api.courier.com/send/"
headers["Authorization"] = "Bearer %s" % (secrets["courier_token"])
headers["Accept"] = "application/json"
headers["Content-Type"] = "application/json"

courier_JSON = {
    "event": "MAGTAG_NOTIFICATION",
    "recipient": "discord_channel",
    "data": {
        "title": response.json()["data"]["blogPost"]["title"],
        "introduction": response.json()["data"]["blogPost"]["introduction"][0:170]
        + "...",
        "url": blog_url,
        "author": author_string,
        "publish_date": strdate,
    },
}

Next, we set a loop to start listening for button press events. If a button is pressed, we display some colors to give feedback that the board is doing something.Then make the request to Courier.

print("Starting the loop, go ahead and press some buttons. :)")
while True:
    if magtag.peripherals.button_a_pressed:
        if button_a_pressed == False:
            for i, b in enumerate(magtag.peripherals.buttons):
                magtag.peripherals.neopixel_disable = False
                magtag.peripherals.neopixels[i] = button_colors[i]
                time.sleep(0.25)
                magtag.peripherals.neopixels[i] = (0, 0, 0)
            print("Making request to Courier")
            response = requests.post(endpoint, json=courier_JSON, headers=headers)
            button_a_pressed = True
            print("Courier response: ")
            print(response.json())

    if magtag.peripherals.button_d_pressed:
        for i, b in enumerate(magtag.peripherals.buttons):
            magtag.peripherals.neopixel_disable = False
            magtag.peripherals.neopixels[i] = button_colors[i]
            time.sleep(0.25)
            magtag.peripherals.neopixels[i] = (0, 0, 0)
            # magtag.peripherals.play_tone(button_tones[i], 0.25)
    else:
        button_a_pressed = False
        magtag.peripherals.neopixel_disable = True
pass

Hit the button to make the board light up and send your first request!

Configuring Courier

Courier supports multiple notification streams including Discord, SendGrid and Twilio. On the Courier website, connect to an integration platform (in my case, I chose Discord) by clicking and following the instructions. For this example, the specific integration you pick doesn’t matter, but you will need to provide API Keys from the service you want to use to Courier.

Next, create a new notification. Since we’ve made a request from the board first, we’ll be able to use the record of that request when setting up our notification.

Select the integration platform you previously used (in my case, Discord), and add it as a channel. From here you’ll be able to design what your notification will look like when it’s delivered. Since we’re passing along the information from the article in our JSON request, we’ll be able to use that data in our notification design.

To wrap things up in the Courier dashboard, head to the Data tag and click on one of the records that matches the event you set in the POST request from the MagTag. Since the request won’t have a notification associated with it at this point, you can click the Map to Notifications and then select the notification you just designed.

Save the record as a test event and head back to the notification designer. Hit preview on the notification you’ve designed, and Courier will use the record to show what the notification will look like when you send it.

Finally head back to your MagTag and hit the button to make a request. The MagTag will make a call to Courier, which will in turn process the request and create a notification to your connected integration platform.

That’s everything the MagTag needs to display content on the screen and send out a link to the full URL if a user hits a button.

If you’ve got a MagTag, head over to GitHub for a link to the full source code with dependencies, an exported content model, instructions on how to set up Courier and a guide on how to install it on your device. Next up for me is to convince our CFO to let me order 559 more MagTags so I can cover the lobby of the Berlin Office with our Contentful version of moveable type. Until then I’ll be keeping my MagTag up on my fridge!

Contentful Changes: Phrase full-text search, GraphQL Richtext

Shy Ruparel — Wed, 10 Feb 2021 17:49:12 +0000

Every month Shy from the Contentful Dev Rel Team shares his favorite changes from the Contentful Changelog. This January he's sharing a visual reference of the Contentful web app, updates to RichText in GraphQL and support for Full Phrase search.