DEV Community: Robert Peterman

Disaster Response Data Sources

Robert Peterman — Thu, 24 Oct 2019 22:35:13 +0000

I recently wrapped up work on a few different disaster related apps and wanted to share some of my thoughts on the data sources I used. I’ll talk about my use cases for each source and discuss other important details like cost and the overall quality/usefulness of the data exposed. Obviously this isn’t an exhaustive list but my hope is that this will help jumpstart someone’s research as they start planning a disaster focused app.

HERE Weather

API Documentation

HERE Weather provides weather forecasts, reports, and detailed information about the current conditions at specific places.
The API can also compile NWS alerts (for US locations) and international weather alerts based on a search query.
Astrology related data can be queried (sun rise, sun set, moon phases).
The most coverage seems to be North America + EU countries.
Other international locations can be limited to current conditions and basic forecast data (no weather alerts).
Overall, HERE Weather features are similar to other major weather related APIs except for its generous freemium plan (250k monthly requests for free).

For one app, I used HERE Weather to get near real time data on a list of cities that were within the general radius of an ongoing disaster. Here’s an example of the endpoint and it’s resulting data.

https://weather.api.here.com/weather/1.0/report.json?app_id={APP ID}&app_code={APP CODE}&product=observation&name=Berlin-Tegel

Result (You'll be able to see the example data after scrolling down the page)

Thanks to the 250,000 monthly calls available (about 8,333 daily) in the free tier of the API, I was able to poll the HERE servers every fairly often from my own server and then forward the updates to the front end of my app resulting in near real time weather data being displayed. I think this is the biggest area where HERE really stands out because many competitors would charge a significant amount to let you poll their servers at higher rates. Getting prototypes off the ground and keeping costs low should be much easier with HERE.

I did run into an issue with the alerts endpoint where I had expected results for a location in Asia but never ended up getting this data after a major storm passed directly through the region. From what I understand the alerts endpoint is much more reliable for US/EU locations so I wanted to mention that since it may not be obvious from the API documentation.

National Weather Service API

API Documentation

Provides weather forecasts, reports, and alerts about the US and its territories
Free to use
Currently uses an authentication string to identify your app - this can be whatever you want, no account creation needed
No published rate limits although users should use best practices and not bombard the system.

HERE Weather and the NWS API overlap a little bit but I wanted to mention the NWS API because it’s a great source of free and reliable data for the US and its territories. While the forecasts and report endpoints are great, in my opinion their alerts endpoint is the most useful feature of the API. For my use case I wanted to get a list of all weather alerts for a given US state.

Example GET request - https://api.weather.gov/alerts/active/area/CA

The above URL will list every active alert in California along with the areas impacted and a text summary of the event. You may need to do a little data cleanup/formatting because some of the raw data can be tough to read but overall this API can give your end users access to some very insightful data.

List of the full range of events the API returns data for

It should also be mentioned that many other weather related APIs incorporate NWS data into their own systems.

USGS Earthquakes

API Documentation

Returns detailed earthquake information from USGS resources
Results can be narrowed down by magnitude, location, time, PAGER level, etc.
CSV, GeoJSON. KML, Quakeml, text, XML, formats
Free to use

Another resource I found to be very useful was the US Geological Survey API. The data can be used as a source for up to date seismic event information and can also be used as a way to query historical earthquake data. Each event that the API returns usually includes a link to a visual summary of the raw data.

Here’s an example

Example of all recent significant earthquakes

During the July 2019 earthquakes in California I ran into an issue where the API suddenly started returning CORS errors. I followed up with the webmasters in charge of the API and they confirmed that this was due to a sudden spike of up to 50k requests a second in the immediate aftermath of the earthquakes. They did mention that work was ongoing to make sure these services remain stable during disasters.

Relief Web

API Documentation

Grants access to Relief Web archives going back to 1970s
Free to use - 1000 calls a day
Endpoints - reports, disasters, countries, jobs, training, sources, faq, blog, book and references.
Bulk of content is humanitarian reports on past and ongoing disasters.

Relief Web in my experience has been a really solid source for in depth summary information that helps give perspective to some of the raw data being displayed elsewhere in my apps. Obviously these reports take time to be written/published so they aren’t always available immediately after a disaster. The data returned by the API is usually linked to other related hrefs that will expose additional data in even more detail.

Here’s a simple example that returns all reports related to Yemen

CIA World Factbook

Mostly static data/statistics
Covers history, people and society, government, economy, energy, geography, communications, transportation, military, and transnational issues for countries
Also includes maps, comparative data, and world flags.
Free but needs to be self hosted in some way

The CIA World factbook is a great resource which can be viewed in web page format here

The only problem is that the CIA does not serve this data through an API. Luckily someone took the time to scrape the data and put it into JSON format.

You can even download the web scraping code to do it yourself or take the easy way and download a copy of the latest data (seems to be updated every 1-2 months). Either way this unofficial API is a great way to allow your users to quickly get up to speed on an impacted region during a disaster. In addition, the world ranking statistics that are available for certain data points help to highlight the vulnerabilities a country might have in a way that other data sources cannot.

ArcGIS

API Documentation

Geographic information system used for creating and interacting with maps,
You can compile, analyze, and share your creations
ArcGIS online includes a large searchable database of public layers/services
Many disaster focused layers that are free to use and updated as often as every 5 minutes

ArcGIS is probably my favorite of this list and is basically a system for displaying data on maps. For my use case I wanted to display weather alerts, earthquake data, forest fires, and other disaster related info onto a map and have some elements of my UI be able to change what location the map centered over. Achieving this did not take a lot of code but I personally felt a bit overwhelmed by the amount of options the API gives you. There are a number of tutorials available to help get you off the ground though and it’s likely that there’s already a tutorial out there for whatever you’re looking to do.

Here’s a list of some of the more popular public layers out there

That’s about it for my favorite disaster related resources. If anyone has any other useful resources please free to share them!

Adding "Dev Data" to your Portfolio

Robert Peterman — Mon, 15 Jul 2019 19:51:32 +0000

When a recruiter checks out your portfolio they’re typically in and out in very quickly. Chances are they’ve got a stack of applicants who all need to get looked at and they simply don’t have the time to spend longer than 1-2 minutes looking at a single portfolio(if that). So it goes without saying that your portfolio needs to stand out to avoid being forgotten in the pile. One way to help a recruiter remember you is to leverage your “developer data” to give them a better sense of who you are as a developer.

Developer Data

Your developer data is basically all of the finger prints you’ve left on the internet while doing dev stuff. Your public commit history, tech blog posts, coding challenges, pull requests, and open source contributions are all great examples. Displaying this data publicly and organized into your portfolio is a great way to show what your skills are and to display that you have verifiable experience with these skills. If you only have a limited amount of time to leave an impression with a recruiter, this is the kind of stuff you want them to see. Below are a few sources of dev data that I wanted to share.

GitHub Data

GitHub API Docs

GitHub is probably the most obvious place to look for developer data so we’ll start there. Thankfully GitHub has a public API and is pretty straight forward if you are just trying to get your commit history and basic user data. Here’s my getGitHubData function from my portfolio site.

// high level wrapper for GitHub API
import GitHub from "github-api";
import axios from "axios";

export default async function getGitHubData() {
  // authenticate with your token - this is your "personal access token" and can
  // be created through your GitHub account dashboard
  var gh = new GitHub({
    token: process.env.GITHUB_TOKEN
  });

  // get profile data
  let me = gh.getUser();
  return me.getProfile().then(userData => {
    return (
      axios
        // get public commits
        .get("https://api.github.com/users/Robp773/events/public")
        .then(events => {
          let filteredCommits = [];

          // events from your GitHub account - these come automatically ordered by most recent
          events.data.map(event => {
            // filtering out events that don't have a commit in their payload
            if (event.payload.commits) {
              event.payload.commits.map(singleCommit => {
                filteredCommits.push({
                  repo: event.repo.name,
                  singleCommit,
                  created_at: event.created_at
                });
              });
            }
          });

          // I also count up my recent commits by repo for a chartjs graph 
          // but I cut that out for simplicity

          return {
            // commit list
            commits: filteredCommits,
            //  user data
            userData: userData.data
          };
        })
    );
  });
}

I eventually use this data on my site to display my commits in a scrollable list below a chart of how my commits break down by repository. I also use the GitHub user data on my home page for my bio and to display how many days I’ve been working as a developer (I basically used my GitHub account creation date as my dev birthday - cheesy I know).

I don’t yet have an impressive list of open source contributions to list but if you do, I’d highly recommend compiling them through the API. This will get you instant brownie points from anyone looking through your portfolio.

DevTo + Dev Blogging APIs

I’ve been trying to make a point of writing articles on tech to both solidify my knowledge on topics and also to show employers that I actually enjoy thinking about code outside normal 9-5 hours. Since I’m mostly using dev.to right now - I used their unofficial API to get my blog articles. Here’s another dev.to article to use as a reference for some of the known endpoints - The state of dev.to v0 api. It looks like an official API has been in the works for a while but has yet to be released. To summarize - https://dev.to/api/articles?username={your user name}&page=0 will get you an array of your blog posts along with comments, reactions, tags, URL, etc.

CodeWars Coding Challenges

CodeWars API Docs

Another way to showcase your skills is by displaying your coding challenge activity. I use CodeWars on my site. My favorite endpoint is /api/v1/users/:id_or_username/code-challenges/completed?page=0 which will return the names of your recently completed challenges along with the time of completion and languages used. For whatever reason, there aren’t many other coding challenge platforms that allow you to access your activity so it might be a struggle getting this data outside of CodeWars. Codewars also gives you access to SVG badges to display your rank and profile info on outside web pages. Here's an example.

Additional Sources

I'll be adding additional sources to this list as I discover them but for now I just wanted to mention Stack Overflow's user API. This is a great place to get data detailing your reputation and badge count and would also help to show your willingness to assist other developers.

Final Thoughts

So here’s what all of this data looks like in my portfolio’s activity page. The UI could use some more work but hopefully you get what I’ve tried to create - a page that a recruiter can quickly look through and instantly get a feel for what I’m doing as a developer. Everything's all in one place so they don’t have to go digging to learn more about me.

The last thing I wanted to mention was the importance of having a fast loading portfolio site since that does play a big role in first impressions. In order to avoid having 5+ loading spinners on my activity page I built my site with React Static + an automated daily webhook from Zapier to keep all of the static data up to date. If you are unfamiliar with static site generators, basically they bundle all of your data into your website’s routes at build time so that your static data is almost instantly available on page load. I’d highly recommend going this route if you plan on pulling in a ton of data for performance reasons. I actually wrote an article on React Static if anyone wants a primer on how to get started with that.

That’s all I’ve got, if anyone has any further ideas for other sources of “developer data” I’d love to hear about them!

React Static Basics

Robert Peterman — Mon, 20 May 2019 19:18:26 +0000

Since I’m a big fan of React, I’m always looking for new React based tools to add to my tool belt. However, there’s been one area of the React ecosystem that I’ve been avoiding… static site generators. A while back I had looked at the documentation for Gatsby to see if it fit my use case for a freelance project. After looking through all the new concepts I would have to learn and getting the sense that it was probably overkill for what I needed to do, I opted for a basic WordPress site and moved on. Thankfully I was recently reintroduced to static site generators through React Static and have been really impressed with how easy it is to get up and running. Not to mention how fast it is.

What is React Static?

According to its GitHub repo, React Static is a progressive static site generator built with a focus on performance, flexibility, and developer experience. Many of its features were built to address the shortcomings of other static site generators like Gatsby and Next.js. Since I’ve never used either of these I can’t go into great detail about how all of these generators differ but at a high level React Static achieves the ease of use of Next.js while coming (somewhat) close to matching the performance of Gatsby. React Static makes sure that creating static sites doesn’t get tedious or overly complex and always maintains a create-react-app feel. These areas are where React Static differentiates itself from competitors.

Getting Started

The developer experience with React Static is one of it’s highlights. If you’ve used “create-react-app” before you’ll appreciate their “react-static create” which will setup a template for you with zero configuration. Run these commands to get started.

npm i -g react-static
react-static create

After running these commands you can choose from a number of different starter templates. For this article we will go with the basic template. After the template has been created for you, open up the new directory and open static.config.js.

Importing data

To summarize, this file's job is to match our project's routes with a page and the data needed by that page. Read through the comment code to get an idea of whats going on.

//static.config.js
export default {

  // resolves an array of route objects 
  getRoutes: async () => {

    // this is where you can make requests for data that will be needed for all
    // routes or multiple routes - values returned can then be reused in route objects below

    // starter template has a request to an endpoint that retrieves an array of fake blog posts
    const { data: posts } = await axios.get(
      "https://jsonplaceholder.typicode.com/posts"
    );

    return [
      // route object
      {
        // React Static looks for files in src/pages (see plugins below) and matches them to path
        path: "/blog",
        // function that returns data for this specific route
        getData: () => ({
          posts
        }),
        // an array of children routes
        // in this case we are mapping through the blog posts from the post variable above
        // and setting a custom route for each one based off their post id
        children: posts.map(post => ({
          path: `/post/${post.id}`,
          // location of template for child route
          template: "src/containers/Post",
          // passing the individual post data needed
          getData: () => ({
            post
          })
        }))
      },
    ];
  },
  // basic template default plugins
  plugins: [
    [
      require.resolve("react-static-plugin-source-filesystem"),
      {
        location: path.resolve("./src/pages")
      }
    ],
    require.resolve("react-static-plugin-reach-router"),
    require.resolve("react-static-plugin-sitemap")
  ]
};

You can then access this data inside the blog page like this.

// src/pages/blog
import React from 'react'
// provides the route's data
import { useRouteData } from 'react-static'
import { Link } from 'components/Router'

// making a ul of links to all of the individual blog posts
export default function Blog() {
  const { posts } = useRouteData()
  return ( 
    <div>
      <h1>It's blog time.</h1>
      <div>
        <a href="#bottom" id="top">
          Scroll to bottom!
        </a>
      </div>
      <br />
      All Posts:
      <ul>
        {posts.map(post => (
          <li key={post.id}>
            {/* linking to individual post routes set by children property of /blog route */}
            <Link to={`/blog/post/${post.id}/`}>{post.title}</Link>
          </li>
        ))}
      </ul>
      <a href="#top" id="bottom">
        Scroll to top!
      </a>
    </div>
  )
}

That's all there is to it - you bring in data, feed it into your routes, and then use that data however you want.

CMS Integration

At some point you might want to hook up your React Static site to a CMS. Since the developer experience is a focus with React Static, we might as well use a CMS that takes this same approach – Netlify CMS. The Netlify CMS is git based so whenever your CMS is updated, the updated content is pushed to your static site’s GitHub repo which in turn triggers Netlify to rebuild the site (and the updated data being added to the appropriate routes through static.config.js).

To set this up, push your static site repo to GitHub and connect it to Netlify by clicking the “New site from Git” button inside your Netlify Dashboard.

Enter "npm run build" for the build command and "dist" for the root directory.

Then go to the Identity tab and select “Enable Identity Service”.

Under Registration preferences select Invite Only.

I’d suggest selecting a provider under “External Providers” for convenience. I’m using Google for this tutorial.

Select the identity tab towards the top of the page and click invite users – invite your personal gmail account to give yourself access. Finish the registration process through your email.

Scroll down to Services > Git Gateway, and click Enable Git Gateway.

Now go back to the root of your project directory and find your public folder – it should already have robots.txt inside. Add an admin and uploads folder here. Inside the newly created admin folder create two new files – config.yml and index.html.

Paste the following code into config.yml. This sets the structure of a collection labeled Test Collection.

backend:
  name: git-gateway
  branch: master # Branch to update (optional; defaults to master)

media_folder: "public/uploads" # Media files will be stored in the repo under public/uploads
public_folder: "/uploads" # Folder path where uploaded files will be accessed, relative to the base of the built site

collections:
  - name: "test-collection" # Used in routes, e.g., /admin/collections/test-collection
    label: "Test Collection" # Used in the UI
    folder: "src/test-collection" # The path to the folder where the documents are stored
    create: true # Allow users to create new documents in this collection
    slug: "{{year}}-{{month}}-{{day}}-{{slug}}" # Filename template, e.g., YYYY-MM-DD-title.md
    fields: # The fields for each document, usually in front matter. 
    # Remove any that aren't needed for posts
      - {label: "Layout", name: "layout", widget: "hidden", default: "blog"}
      - {label: "Title", name: "title", widget: "string"}
      - {label: "Publish Date", name: "date", widget: "datetime"}
      - {label: "Body", name: "body", widget: "markdown"}

Then paste this code into into the index.html file. This sets up the admin login modal + CMS at the /admin route of your site.

<!doctype html>
<html>
<head>
  <meta charset="utf-8" />
  <meta name="viewport" content="width=device-width, initial-scale=1.0" />
  <title>Content Manager</title>
  <script src="https://identity.netlify.com/v1/netlify-identity-widget.js"></script>
</head>
<body>
  <!-- Include the script that builds the page and powers Netlify CMS -->
  <script src="https://unpkg.com/netlify-cms@^2.0.0/dist/netlify-cms.js"></script>
  <script>
    if (window.netlifyIdentity) {
      window.netlifyIdentity.on("init", user => {
        if (!user) {
          window.netlifyIdentity.on("login", () => {
            document.location.href = "/admin/";
          });
        }
      });
    }
  </script>
</body>
</html>

Last but not least navigate back to static.config.js and add this code. Once we add a post to our "Test Collection" collection this function will read it from our repo and make it available to our route objects. You'll need to run npm install fs klaw gray-matter to download the dependencies.

//static.config.js

//...other imports

//for reading local files
import fs  from "fs";
import klaw from "klaw";
import matter from "gray-matter";

function getPosts() {
  const items = [];
  // Walk ("klaw") through posts directory and push file paths into items array //
  const getFiles = () =>
    new Promise(resolve => {
      // Check if test-collect directory exists //
      // This is the folder where your CMS collection we made earlier will store it's content. Creating a post inside this collection will add a "test-collection" directory to your repo for you.
      if (fs.existsSync("./src/test-collection")) {
        klaw("./src/test-collection")
          .on("data", item => {
            // Filter function to retrieve .md files //
            if (path.extname(item.path) === ".md") {
              // If markdown file, read contents //
              const data = fs.readFileSync(item.path, "utf8");
              // Convert to frontmatter object and markdown content //
              const dataObj = matter(data);
              // Create slug for URL //
              dataObj.data.slug = dataObj.data.title
                .toLowerCase()
                .replace(/ /g, "-")
                .replace(/[^\w-]+/g, "");
              // Remove unused key //
              delete dataObj.orig;
              // Push object into items array //
              items.push(dataObj);
            }
          })
          .on("error", e => {
            console.log(e);
          })
          .on("end", () => {
            // Resolve promise for async getRoutes request //
            // posts = items for below routes //
            resolve(items);
          });
      } else {
        // If src/posts directory doesn't exist, return items as empty array //
        resolve(items);
      }
    });
  return getFiles(); 
}

Add const test = await getPosts() inside getRoutes next to where we make the blog posts request. This test variable will now hold the values of any content we make in our collection.

Create a new route object and add it to the array of routes like so...

// add below /blog route object
      {
        path: "/test",
        getData: () => ({
          test
        }),
        children: test.map(post => ({
          // actual path will be /test/"whatever the post slug is"
          path: `/${post.data.slug}`,
          // location of template for child route
          template: "src/containers/Test-Post",
          // passing the individual post data needed
          getData: () => ({
            post
          })
        }))
      }

At this point we just need a page to display our content to make sure everything is working. To accomplish this quickly, just add a test.js file to src/pages and paste this code. We're creating a list of links to each test-collection post we've made.

// src/pages/test.js

import React from "react";
import { useRouteData } from "react-static";
import { Link } from 'components/Router'

export default function Test() {
  const { test } = useRouteData();
  return (
    <ul>
      {test.map((item, index)=>{
          return (
              <li key={index}>   
                <Link to={`/test/${item.data.slug}`}>{item.data.title}</Link>
              </li>
          )
      })}
    </ul>
  );
}

Then add in your new post template by creating Test-Post.js in the containers folder.

// src/containers/Test-Post.js

import React from 'react'
import { useRouteData } from 'react-static'
import { Link } from 'components/Router'
// very similar to Post.js component
export default function Post() {
  // get the post data
  const { post } = useRouteData()
  return (
    <div>
      <Link to="/test/">{'<'} Back</Link>
      <br />
      {/* print out what we want to display */}
      <h3>{post.data.title}</h3>
      <div>{post.content}</div>
    </div>
  )
}

Also add a nav link to the new page in src/App.js for convenience <Link to="/test">Test</Link>

After pushing your changes to your GitHub repo, Netlify will now rebuild the site with Netlify CMS included. With your live Netlify url open, add /admin to the root url of your site in the browser to login to Netlify CMS. Once you’re in, write up some content and publish it inside "Test Collection". Netlify will once again rebuild your site with the new content you’ve added. When the rebuild is done your test.js page that was created earlier should display all the links to your posts.

That’s all there is to it. You now have a lightning fast static site connected to a modern CMS with a fully automated deployment flow.

Final Repo

Note – if you see two 404 errors in the developer console when using the Netlify CMS you can safely ignore these. I confirmed with some Netlify Community members that these are expected errors and are caused when you do not use Git LFS / Netlify Large Media settings. This issue is actually marked as good first issue if anyone is interested - https://github.com/netlify/netlify-cms/issues/2158