Sascha

Posted on Sep 10, 2020 • Edited on Sep 28, 2020

Get Historical Stats for your Dev.to Articles

#showdev #tutorial #javascript #datascience

Recently, I posted my first article on Dev.to. As expected there was an initial wave of some views which eventually stopped. Two weeks later I posted a second article. It was related to the first article and had a link to it. As expected, the views of my first post went up again. Since I love data, I wanted to see how many views belonged to the first wave and how the views on my articles change based on the what else I publish or other events that might happen. Unfortunately, Dev.to only shows me the final view count. To still my data hunger, I created a little app...

Feel free to visit the running app for my stats, or check out the code repository to dive into the code immediately.

TL;DR (for the efficient and impatient)

You want to deploy the app for your own Dev.to stats? You don't have time to read step by step tutorials? Well, here you go...

Deploying the app to Heroku only takes a minute but before you can do this, you need to do the following:

If you don't have one already, create a Heroku account and install their CLI
Go to RethinkDB Cloud and request free alpha access to the RethinkDB Cloud add-on.
Get an API key from Dev.to (Settings → Account → DEV API Keys)

Now run:

$ git clone git@github.com:mostlytyped/dev.to-stats.git
$ cd dev.to-stats/
$ heroku create
$ heroku addons:create rethinkdb
$ heroku config:set API_KEY=<YOUR_DEV_TO_API_KEY>
$ git push heroku master
$ heroku open

Note: the collection worker needs to be enabled manually on the apps resources dashboard. Also, free Heroku apps are put to sleep when they are inactive so the stats collector will not run regularly unless you pay for a dyno. Alternatively, you can visit your site once a day. For now, this is what I do ;-)

That's it, you are done. Enjoy your historical stats.

A step by step tutorial

Going fast is great but how about learning new skills and tools? In this section you will learn how to create the app from the ground up. In particular, you will...

... learn how to use RethinkDB, the totally awesome document database. It is like MongoDB but has reactivity built. This allows you to subscribe to queries. Oh, and it is still open source!
... create an embedded Vue.js app. That is a Vue app that you don't have to compile. I bet you haven't done that before.
... use Chart.js to plot the stats. It is always useful to have a plotting library in your tool kit.

Application setup

We will build a Node.js app, so you need to have node and npm installed. If you want to deploy your app to Heroku, you will also need a Heroku account, as well having their CLI installed. To run your app locally, you need to install and run a RethinkDB instance.

To create the application, run the following in a terminal.

$ mkdir devto-stats && cd devto-stats
$ npm init -y
$ npm install rethinkdb express morgan axios

This will initialize a Node.js app and install all required dependencies.

Prepare a Heroku app

In order to deploy the application to Heroku we need to create a Heroku app:

$ git init
$ heroku create

We will also need a RethinkDB instance to store articles and their daily stats. You can do this via the RethinkDB Cloud add-on as follows:

$ heroku addons:create rethinkdb

The RethinkDB Cloud add-on is currently in alpha. Request an invite for your Heroku account email.

Get the Dev.to API key

To access the stats of your articles you need an API key from Dev.to. You can get one under Settings → Account → DEV API Keys. Add the key to your Heroku app:

$ heroku config:set API_KEY=<YOUR_DEV_TO_API_KEY>

Collect the stats

To collect the stats we basically need to do two things on repeat: (i) get the stats for you articles from Dev.to and (ii) save the stats to RethinkDB. We need to run the stats collection at least every 24h to make sure we get stats once per day (Dev.to only updates the stats once daily).

// collect.js

const axios = require("axios");
const r = require("rethinkdb");
const { getRethinkDB } = require("./reql.js");

// Get articles from Dev.to
// ...

// Save article stats to RethinkDB
// ...

// Run once immediately
saveStats();
// Interval should be less than 24h. Running more than once a day
// is not a problem but a missed day cannot be recovered.
const interval = 6 * 60 * 60 * 1000; // 6h
setInterval(saveStats, interval);

To get the stats we run a simple axios request. Since the articles
are paged we query new pages until we get one that is not
full. The API_KEY environment variable contains your Dev.to
API key.

// collect.js
// ...

// Get articles from Dev.to
const getArticles = async function () {
  let articles = [];
  let page = 1;
  while (true) {
    let articles_page = await axios.get(
      "https://dev.to/api/articles/me?page=" + page,
      {
        headers: {
          "api-key": process.env.API_KEY,
        },
      },
    );
    articles.push(...articles_page.data);

    // If a page is not full we are done
    if (articles_page.data.length < 30) {
      break;
    }
  }
  return articles;
};

// ...

When saving the stats for the day we first need to check if the article already exists in our database. If not we add it. Then we save the stats as long as we have not done so already today.

// collect.js
// ...

// Save article stats to RethinkDB
const saveStats = async function () {
  const now = new Date();
  let day = ("0" + now.getDate()).slice(-2);
  let month = ("0" + (now.getMonth() + 1)).slice(-2);
  let year = now.getFullYear();
  const today = year + "-" + month + "-" + day;
  console.log("Collect stats:", today);

  // Get all articles
  const articles = await getArticles();

  // Save stats
  let conn = await getRethinkDB();
  articles.forEach(async (article) => {
    let db_article = await r.table("articles").get(article.id).run(conn);
    if (!db_article) {
      // New article -> save
      await r
        .table("articles")
        .insert({
          id: article.id,
          title: article.title,
          url: article.url,
          latest_stats: today,
        })
        .run(conn);
      // Save stats
      await r
        .table("stats")
        .insert({
          article_id: article.id,
          date: today,
          comments: article.comments_count,
          reactions: article.public_reactions_count,
          views: article.page_views_count,
        })
        .run(conn);
    } else if (db_article.latest_stats < today) {
      // Existing article -> update
      await r
        .table("articles")
        .get(article.id)
        .update({ latest_stats: today })
        .run(conn);
      // Save stats
      await r
        .table("stats")
        .insert({
          article_id: article.id,
          date: today,
          comments: article.comments_count,
          reactions: article.public_reactions_count,
          views: article.page_views_count,
        })
        .run(conn);
    } else {
      console.log("Already got stats today for article " + article.id);
    }
  });
};

// ...

As you might have noticed get the RethinkDB connection from reql.js. Let's implement this now.

Handling the RethinkDB connection

Connecting to RethinkDB is straight-forward. We only add a bit of logic to handle disconnections gracefully. The RETHINKDB_* environment variables will be set automatically by the RethinkDB Cloud add-on. The defaults work for a locally running RethinkDB instance.

// reql.js

const r = require("rethinkdb");

let rdbConn = null;
const rdbConnect = async function () {
  try {
    const conn = await r.connect({
      host: process.env.RETHINKDB_HOST || "localhost",
      port: process.env.RETHINKDB_PORT || 28015,
      username: process.env.RETHINKDB_USERNAME || "admin",
      password: process.env.RETHINKDB_PASSWORD || "",
      db: process.env.RETHINKDB_NAME || "test",
    });

    // Handle close
    conn.on("close", function (e) {
      console.log("RDB connection closed: ", e);
      rdbConn = null;
    });
    // Handle error
    conn.on("error", function (e) {
      console.log("RDB connection error occurred: ", e);
      conn.close();
    });
    // Handle timeout
    conn.on("timeout", function (e) {
      console.log("RDB connection timed out: ", e);
      conn.close();
    });

    console.log("Connected to RethinkDB");
    rdbConn = conn;
    return conn;
  } catch (err) {
    throw err;
  }
};
exports.getRethinkDB = async function () {
  if (rdbConn != null) {
    return rdbConn;
  }
  return await rdbConnect();
};

Building the server

The server is a simple Express.js app that serves a static frontend from the public directory. The server listens to get requests for one route (/article_stats) and returns an array of articles and their stats.

// index.js

// Express app
const express = require("express");
const app = express();

// Logging middleware
const morgan = require("morgan");
app.use(morgan("combined"));

// Serve frontend
app.use(express.static("public"));

// Lazy RethinkDB connection
const r = require("rethinkdb");
const { getRethinkDB } = require("./reql.js");

// Route to get stats
app.get("/article_stats", async (req, res) => {
  const conn = await getRethinkDB();
  let article_cursor = await r.table("articles").run(conn);
  let articles = await article_cursor.toArray();
  let article_stats = await Promise.all(
    articles.map(async (article) => {
      let stats_cursor = await r
        .table("stats")
        .filter({ article_id: article.id })
        .orderBy(r.asc("date"))
        .run(conn);
      let stats = await stats_cursor.toArray();
      article.stats = stats;
      return article;
    }),
  );
  res.json(article_stats);
});

// Start server
const listenPort = process.env.PORT || "3000";
app.listen(listenPort, () => {
  console.log(`Listening on ${listenPort}`);
});

Building the frontend

For our frontend we will use an embedded Vue.js app. This makes the frontend simple, but gives us access to all of Vue's awesome features. The frontend consists of a layout file as well as JavaScript and CSS assets.

HTML layout

The layout file only serves as a mount point for the Vue app in addition to importing the dependencies and assets.

<!-- public/index.html -->

<!DOCTYPE html>
<html lang="en">
  <head>
    <meta charset="UTF-8" />
    <meta name="viewport" content="width=device-width, initial-scale=1.0" />
    <title>Dev.to Historical Stats</title>
    <link href="/css/main.css" rel="stylesheet" />
  </head>

  <body>
    <div id="app"></div>
    <script src="https://cdn.jsdelivr.net/npm/vue/dist/vue.js"></script>
    <script src="https://cdn.jsdelivr.net/npm/chart.js@2.8.0"></script>
    <script src="/js/app.js" language="javascript"></script>
  </body>
</html>

Style sheet

The CSS asset mostly contains the styling for the article table. It is nothing fancy.

/* public/css/main.css */

#app {
  margin: auto;
  width: 80%;
}

#articles {
  font-family: "Trebuchet MS", Arial, Helvetica, sans-serif;
  border-collapse: collapse;
  width: 100%;
}

#articles td,
#articles th {
  border: 1px solid #ddd;
  padding: 8px;
}

#articles tr:nth-child(even) {
  background-color: #f2f2f2;
}

#articles tr:hover {
  background-color: #ddd;
}

#articles th {
  padding-top: 12px;
  padding-bottom: 12px;
  text-align: left;
  background-color: #5c9ead;
  color: white;
}

The Vue app

The JavaScript asset app.js contains the actual Vue app. It consists of a simple component with a Chart.js canvas and an article table in the template. When the component gets created we will get the stats data from the server and create the actual Chart.js chart.

// public/js/app.js

// Vue App
const App = Vue.component("app", {
  data() {
    return {
      articleStats: [],
      chart: {},
    };
  },
  async created() {
    /* Get stats data */
    // ...
    /* Create Chart.js plot from data */
    // ...
  },

  template: `{% raw %}
  <div id="app">
    <div>
        <canvas id="chart"></canvas>
    </div>
    <table id="articles">
      <tr>
        <th></th>
        <th>Article</th>
        <th>Views</th>
        <th>Reactions</th>
      </tr>
      <tr v-for="article in articleStats">
        <td :style="{'background-color': article.color, width: '10px'}"></td>
        <td><a :href=article.url class="title">{{ article.title }}</a></td>
        <td>{{ article.stats[article.stats.length - 1].views }}</td>
        <td>{{ article.stats[article.stats.length - 1].reactions }}</td>
      </tr>
    </table>
  </div>{% endraw %}
      `,
});

// Mount Vue app
var app = new Vue({
  render: (h) => h(App),
}).$mount("#app");

We fetch the article stats from the /article_stats route on the server. In addition we add a random color to each article that we will use for the line in the chart.

// public/js/app.js
// ...

/* Get stats data */

// Fetch article stats from server
const url = new URL(
  document.location.protocol + "//" + document.location.host + "/article_stats",
);
const articleStatsResp = await fetch(url);
let articleStats = await articleStatsResp.json();

// Assign random color to article
const randomColor = function () {
  var r = Math.floor(Math.random() * 255);
  var g = Math.floor(Math.random() * 255);
  var b = Math.floor(Math.random() * 255);
  return "rgb(" + r + "," + g + "," + b + ")";
};
articleStats.forEach((article) => {
  article.color = randomColor();
});
this.articleStats = articleStats;

// ...

Now we need to transform the stats into a Chart.js config object. We will do so in three steps:

We need the x-axis labels. For this we will use the date fields from the longest stats array of all articles (oldest article).
Then we transform the article stats into datasets Chart.js can plot. Most importantly we need to prepend 0 values to the stats array of newer articles to make sure they are all the same length.
Create a Chart.js config object with all the display options we want.

Once we have the Chart.js config object we create a new chart and mount it in the designated HTML canvas element.

// public/js/app.js
// ...

/* Create Chart.js plot from data */

// Get x-Axis labels
let labels = [];
let minDate = "9"; // This will work for the next ~8000 years
this.articleStats.forEach((article) => {
  if (article.stats[0].date < minDate) {
    minDate = article.stats[0].date;
    labels = article.stats.map((stat) => {
      return stat.date;
    });
  }
});

// Transform article stats into Chart.js datasets
let datasets = this.articleStats.map((article) => {
  let data = [];
  // Fill with 0 until first view
  for (let date of labels) {
    if (date >= article.stats[0].date) {
      break;
    }
    data.push(0);
  }
  // Append views
  data.push(
    ...article.stats.map((stat) => {
      return stat.views;
    }),
  );
  // Return data set for this article
  return {
    label: article.title,
    data: data,
    fill: false,
    borderColor: article.color,
    backgroundColor: article.color,
  };
});

// Chart.js config
let chartConfig = {
  type: "line",
  data: {
    datasets: datasets,
    labels: labels,
  },
  options: {
    responsive: true,
    // aspectRatio: 3,
    title: {
      display: true,
      text: "Dev.to Article Stats",
    },
    legend: {
      // Our table is the legend
      display: false,
    },
    tooltips: {
      mode: "index",
      intersect: false,
    },
    hover: {
      mode: "nearest",
      intersect: true,
    },
    scales: {
      xAxes: [
        {
          display: true,
          scaleLabel: {
            display: true,
            labelString: "Date",
          },
        },
      ],
      yAxes: [
        {
          display: true,
          scaleLabel: {
            display: true,
            labelString: "Views",
          },
        },
      ],
    },
  },
};

// Create chart
let ctx = document.getElementById("chart").getContext("2d");
this.chart = new Chart(ctx, chartConfig);

// ...

Now we have the fronend and the server that serves it. Before we can deploy and run our app we only need a migration script to create the actual tables in the database.

Database migration

The app does not work without a articles and stats tables. We thus need a database migration that adds these.

// migrate.js

var r = require("rethinkdb");

r.connect(
  {
    host: process.env.RETHINKDB_HOST || "localhost",
    port: process.env.RETHINKDB_PORT || 28015,
    username: process.env.RETHINKDB_USERNAME || "admin",
    password: process.env.RETHINKDB_PASSWORD || "",
    db: process.env.RETHINKDB_NAME || "test",
  },
  async function (err, conn) {
    if (err) throw err;
    console.log("Get table list");
    let cursor = await r.tableList().run(conn);
    let tables = await cursor.toArray();

    // Check if articles table exists
    if (!tables.includes("articles")) {
      // Table missing --> create
      console.log("Creating articles table");
      await r.tableCreate("articles").run(conn);
      console.log("Creating articles table -- done");
    }

    // Check if stats table exists
    if (!tables.includes("stats")) {
      // Table missing --> create
      console.log("Creating stats table");
      await r.tableCreate("stats").run(conn);
      console.log("Creating stats table -- done");
      // Create index
      await r
        .table("stats")
        .indexCreate("article_date", [r.row("article_id"), r.row("date")])
        .run(conn);
      console.log("Creating article-date secondary index -- done");
    }

    await conn.close();
  },
);

This migration checks if the tables exists and creates them if they are missing. For the stats table we will also create a secondary index to make sure there is only ever one stats document for the same article_id and date.

Deploy the application to Heroku

To deploy our working application to Heroku we need to create a Procfile. This file basically tells Heroku what processes to run.

// Procfile

release: node migrate.js
web: node index.js
collect: node collect.js

The release and web processes are recognized by Heroku as the command to run upon release and the main web app respectively. The collect process is just a worker process that could have any name.

Deploy the app to Heroku with

$ echo "node_modules/" > .gitignore
$ git add .
$ git commit -m 'Working dev.to stats app'
$ git push heroku master

You will need to manually enable the collect process in your Heroku app. You can do so on the Resources tab.

Wrapping up

With this app running on Heroku I can finally go back to coding and writing articles without missing any move my article stats might be making.

Let me know if this app is useful to you, if there are any bugs, or if there are any features you would want me to add.

DEV Community