DEV Community: megazear7

Weather Bear!

megazear7 — Sun, 21 Mar 2021 09:53:59 +0000

I made a weather app... Check it out at https://weatherbear.app

I used LitElement for building components, some Netlify Lamdas for the data, and built it in a SPA / PWA style.

Introducing JTL: JSON Templating Language

megazear7 — Fri, 23 Oct 2020 11:23:09 +0000

What if you could convert json into html? This would allow you to use json as a templating language for html. Alternatively, you could map json data from an api into html markup. Now you can with JTL: JSON Templating Language.

This was a small pet project of mine; something that I thought might be useful down the road. I'm imaging using JTL as an alternative for html templating languages such as lit-html, handlebars, and the like. Instead of writing a component or controller that then gets rendered into html with a template, you simple maintain a json state representation of your component and render the json into html with JTL.

The easiest way to see JTL in action is by visiting the documentation site at jtl.alexlockhart.me. It can be installed with npm using the @megazear7/jtl package and the code is open source at github.com/megazear7/jtl.

A JTL example

Let's take an example use case to see how you might use JTL. We want to get a list of links from an API and allow the user to add a new link to the list. We won't consider saving the new links to the API, but it will be obvious enough how that could be done as well. To start with we need an HTML page with a custom web component that can serve as the starting point for our example.

<!DOCTYPE html>
<html>
<body>
  <jtl-example-api></jtl-example-api>
</body>
</html>

The web component

Now we need to implement the shell of this web component. We will add a render method which updates the html in the component. Inside of this render method we use JTL to create an empty ul inside of a div.

import { jtl } from '@megazear/jtl';

class ExampleApiComponent extends HTMLElement {
    constructor() {
        super();
        render();
    }

    render() {
        this.innerHTML = '';
        this.appendChild(jtl({
            name: "div",
            children: [
                {
                    name: "ul",
                }
            ]
        }).toHtmlElement());
    }
}

window.customElements.define('jtl-example-api', ExampleApiComponent);

Fetching data from the API

Next we want to pull in data from an API. To do this, we update the constructor to fetch data from a json endpoint and update the state of our component before calling the render method.

constructor() {
    super();
    fetch('/data.json')
        .then(res => res.json())
        .then(json => this.movies = json.movies)
        .then(() => this.render());
}

The render method

Now, we need to update our render method to use the state stored in the movies field. This is where most of the magic of JTL is happening. Notice that all we need to do is take the json from the API and map it into a format that JTL expects. We also add input fields for the title and link as well as a button for adding the new movie link. After rendering the component we then add event listeners for the add and remove buttons.

render() {
    this.innerHTML = '';
    this.appendChild(jtl({
        name: "div",
        children: [
            {
                name: "ul",
                children: this.movies.map(movie => ({
                    name: "li",
                    children: [
                        {
                            name: "a",
                            attrs: { href: movie.imdb },
                            content: movie.title
                        }, {
                            name: "span",
                            attrs: { class: "remove", "data-movie": movie.title },
                            content: " x"
                        }
                    ]
                }))
            }, {
                name: "input",
                attrs: { class: "add-title", placeholder: "Title" }
            }, {
                name: "input",
                attrs: { class: "add-imdb", placeholder: "IMDB" }
            }, {
                name: "button",
                attrs: { class: "add-btn" },
                content: "Add"
            }
        ]
    }).toHtmlElement());

    this.querySelector('.add-btn').addEventListener('click', () => this.addMovie());
    this.querySelectorAll('.remove')
        .forEach(btn => btn.addEventListener('click', (e) => this.removeMovie(e.target.dataset.movie)));
}

The add and remove methods

Our final step is to implement the add and remove events. To do this all we need to do is update our state object and then call the render method. This is an example of functional UI, since our UI is simply a function of our state. In this case the render method is the function that defines the UI in terms of the this.movies object.

class ExampleApiComponent extends HTMLElement {
    ...

    addMovie() {
        const title = this.querySelector('.add-title').value;
        const imdb = this.querySelector('.add-imdb').value;

        this.movies.push({ title, imdb });
        this.render();
    }

    removeMovie(movieTitle) {
        this.movies = this.movies.filter(movie => movie.title !== movieTitle);
        this.render();
    }
}

The final result

Below is a screen shot of the final result. You can see this example in action at jtl.alexlockhart.me/examples/api.html. If you want to make this same example a reality, you will need to create the HTML file and JS modules as shown above and then serve this along with the data.json file. You can see the full implementation on Github and an example of the data.json file at jtl.alexlockhart.me/data.json.

Conclusions

JTL provides a reliable way to turn data into json and fills the role that is traditionally served by html templating languages. It is essentially a different approach to html templating, focusing on the data format (json) instead of focusing on the display format (html). Now that you have seen an example use of JTL, it might be the time to go look at the implementation.

One issue I see with it is performance, since any update to the data requires rerendering the html. Currently you need to remove all the html from the containing element and rerender the whole dom tree. However, since we have the power of json at our fingertips, I am wondering if there would be some way to diff the json, and then only make the updates that are needed? A project for another time.

My goal for JTL was to do something that felt creative and fresh that might inspire a different way of thinking. What ideas does JTL spawn for you? Is it pointless? Does it give you any ideas on alternative approaches to UI? Let me know your thoughts in the comments.

Check out my blog for more of my musings upon technology and various other topics.

Static Websites Can Be Progressive Single Page Applications

megazear7 — Thu, 28 May 2020 13:31:05 +0000

The word 'static' often has negative connotations. It can be used to describe something as limited and rigid. A 'static website' is often used to describe a website that is small, simple, and does not do much. Lets take a second look at this, find out what 'static' really means when it comes to the web, and find if those limitations are true.

A static website is actually just a website where the views are pre-rendered. Let's define our terms. First, a view is simply a chunk of HTML. This could refer to an entire webpage, or a section of a page. To render a view means to take some data and plug it into some sort of template in order to generate the HTML. So, to pre-render a view means to combine the data and the template to generate the HTML before. But before what? In this case it means before the user requested the page.

A walk through history

Historically (you know, in the ancient times) this was the only way the web worked. The world wide web was a collection of linked documents. These documents came in the form of HTML, were identifiable by a URI, and contained links to other documents. All of these documents existed long before you the user ever typed in a URL to go to one. They were either created individually by hand or with some sort of static site generator.

Dynamic rendering

Then came the evolution of the web with server side and client side rendering. These have been the default method of building websites for a long time. This is where you wait for a user to request a webpage and instead of a static document being returned, some sort of framework renders the view. This could be done on the server or in the browser. It might hit databases or user profiles in order to get the data to render the view.

This system is powerful because it is dynamic. Views are only rendered after the user has requested it. Ruby on Rails, Spring, React, Angular, and most of the other web frameworks use this approach. In the history of the web this dynamic system evolved with the concept of single page applications. This meant that as users navigate through the different views of your application, instead of going to new pages for each view, the view was updated 'right from under their feet'. This means that JavaScript DOM APIs are used to update the HTML without navigating to a new URL. This lead to dynamic and interactive experiences. Instead of waiting on page reloads after every click, the website now felt like using a native app.

Progressive web apps

Another evolution came with progressive web apps. Browsers gave websites the ability to have their views cached for offline use and to be installed as if they were native apps. Now users can interact with a website as an app, and never know or even care whether it is a native app or a web app. The only difference is where they installed it from. Native apps are installed from an app store and web apps are installed by visiting a website. However, in coming years even this might change, as app stores from Google and Microsoft might automatically make progressive web apps installable directly from the store.

All of this evolution occurred over the years with static site generation being left as a relic of the past, at least in the minds of many. However this need not be the case. There is nothing fundamentally at odds between static site generation and progressive, single page web applications. It might feel like single page applications require heavy and powerful frameworks, but it does not. It might feel like a progressive web app cannot be achieved with statically generated views, but it can.

A simple reintroduction

If we are to continue our walk through the history of the web, I would add JAMstack as the next destination in our journey. It is really no evolution at all, but rather the attempt to brush the dust off the so called 'relic' which is static site generation, and let the old timer meet the fresh faces of single page applications and progressive web apps. The idea is a simple one; simply render your markup ahead of time and enhance this with JavaScript that connects to APIs. This is nothing crazy or new. It is just reintroducing some long standing methods of building web sites with some other newer concepts.

So what are the benefits of static site generation over the dynamic options of client and server side rendering? Mostly performance, but also security, simplicity, and scalability. In a single page application you need lots of views. This might be things like navigation elements, product entries, profiles, articles, and more.

Instead of each web page being it's own view, the views are split into more fine grained pieces and then combined by the single page application to create full web pages. This is usually done with client side rendering; be it Angular, React, or something similar. However, there is no reason that this cannot be done with the views rendered ahead of time. For example, why wait until the user clicks 'add a product' to render the cart entry view? This can be rendered ahead of time and plugged into the single page application when needed.

This makes the system more performant as you are simple delivering html documents over the network without any server or client side application rendering the view. This also makes it more scalable, since all you have to do is deploy your views to a CDN with no need for some heavy duty web server. In fact, you need no web server at all. Just deploy your views 'to the edge' on a CDN. At the same time this reduces your applications surface area to attack, since there is no web server to hack, just views to retrieve. Of course you might have API integrations which require security, but the point is that you are minimizing the ways in which the system can be exploited to the bare minimum.

Selling the concept

All of this means that a static website is quite dynamic, which does not indicate a contradiction in the design but an ambiguity in terms. Maybe 'static site generation' is not the best phrase to describe a dynamic, fully featured web application. How would we market this approach? What should it be called? JAMstack is one answer. However, to be a bit more explicit and verbose, we could call this a progressive, pre-rendered, single page application.

Finally, I would like to leave you with a small treat. Check out samplejam.alexlockhart.me to see a small example of this in action. The pre-rendered views are built with OrisonJS and deployed to a CDN with Netlify. A small amount of JavaScript provides the single page application and progressive web app features. Additionally, a sample API integration is shown by adding products to your cart. All of this is done without any HTML being rendered at runtime.

Thanks for staying with me through this tour of web history. I hope it has given you some context and motivation to check out the benefits of pre-rendering your views!

Check out my blog for more of my musings upon technology and various other topics.

The Golden Jam of Web Stacks

megazear7 — Wed, 24 Jul 2019 17:37:24 +0000

Over the course of time my go to tech stack for web development has changed. At one time it was Ruby on Rails with Heroku, then Firebase and React, then Polymer and lit-html. While all of these had their place and still work brilliantly I have been able to fine tune my go to technologies; finding a more specific niche use case while also broadening capabilities. As I see it these are the major problems to solve in web development:

Rendering
Hosting
Source control
Content management
Business logic
Data storage
Delivery

Every tech stack will need to have an answer for these problems. The set of technologies that I use falls under the JAMstack paradigm. Here they are listed in same order as the problems they solve:

Orison
Netlify
GitHub
Contentful
Netlify functions
Firebase Firestore
SPA + PWA

The beauty of JAMstack is that any one of these seven parts could be replaced by another technology which solves the same problem. On any given project I might use Gatsby instead of Orison, Firebase Hosting instead of Netlify, Bitbucket instead of GitHub, Blogger instead of Contentful, and the list goes on. As time goes by pieces of the stack can be swapped without major dependencies upon the other layers. The other benefit is that the performance and security of this stack is largely not based upon these technologies themselves, but upon the web platform itself.

While my previous go to tech stacks required lots of specialized knowledge, this tech stack runs very closely to the web platform itself and has precanned solutions to almost all of the major problems of web development.

Redeeming the Time

megazear7 — Fri, 19 Jul 2019 00:25:50 +0000

I recently found myself giving time management advice. During this discussion the phrase, "reducing wasted time, redeeming necessary time" came up. I think this encapsulates the two main methods for increasing productivity. Some time syncs are simply wasteful while others are necessary and yet unproductive. In order to increase productivity wasted time needs to be reduced while necessary time syncs need to be redeemed.

However what is wasted time, or for that matter unredeemed time? Firstly, it assumes a goal. If time is wasted then there is an implicit assumption that it failed to do something. So in order for the concept of wasted time to be meaningful we need to have a plan, process, or goal that we are striving towards. When unplanned and unaccounted for interruptions break into those efforts, we can count that time as being wasted. However let's come down out of the theoretical clouds and talk about it with concrete examples.

The easiest example of wasted time are addictive phone apps; infinite scroll through Facebook or YouTube anyone? It truly is infinite, I can promise you that much (although I have not been able to prove that yet... but am surely coming close). Generally speaking we do not "plan" on scrolling through social media, or some other form of app, for an hour after dinner, or before leaving the gym, or where ever else we do this. Most often this is time unplanned for, unaccounted for, unproductive, dragging us off course, and preventing us from reaching our goals. While it is not in the scope of my thoughts here to give detailed advice on how to overcome this specific example, the point remains clear. In order to be productive we must actively put measures in place to reduce wasted time.

What is redeemed time? The word, "redeem" is not often in our vocabulary, so let us at least see what Old Webster has to say. It lists many definitions, however two that particularly stands out are, "to get or win back" and "to make worthwhile". These two definitions are a perfect marriage for our phrase, "redeem the time". The former assumes that something has been taken, and calls us to action to win it back. The second assumes that while necessary, it is not worthwhile unless we make it so. I have drifted into the stratosphere again, so let's parachute back down and stick the landing on the good ol' concrete.

Say you are driving to work, to the gym, to the in law's, or whatever else has you in your car. Did you plan this time? Or is it sitting there waiting for you to win it back and make it worthwhile? Sure, the destination might be worthwhile, but the time spent in the car, in and of itself without considering the destination, is it worthwhile? Or does it need won back? To do this we could have a podcast play list lined up; something we planned for, something we intentionally decided ahead of time that would be worth our time, not whatever addictive suggestion in the feed has lined up next. You must decide in advance what is worth your time so that you can redeem it.

Check out my blog for more of my musings upon technology and various other topics.

The Sales Pitch of Static Site Generation

megazear7 — Thu, 18 Jul 2019 16:37:53 +0000

I have recently developed a static site generator tool called Orison and for this reason among others I have been trying to "sell" people on static site generators; developers, architects, and non technical folks included. For many people the term "Static Site Generation" is synonymous with "small blog", "personal site", and "trivial use case" instead of phrases like "enterprise scale" and "unbeatable performance".

I am very thankful for everything that Netlify has done to popularize static site generation. However I wander if the naming of the term turns people away. In technical articles, presentations, blog posts, and in person discussion I often use the term "build tool", "build process" or something of that sort to describe the role that the static site generator plays in the JAMstack architecture.

I am curious as to what you have found. Is there a better term that I am missing? Should we instead focus on rebranding the term "static site generator"? Does it even matter?

Thank you ahead of time for any responses. The subjective human experience is often an under realized blocker to technology and especially in this case one that I think would be worth discussing.

Using Async / Await with Callbacks

megazear7 — Mon, 15 Jul 2019 20:46:19 +0000

I recently ran into the problem of getting a promise interface working alongside a callback interface. Specifically I had an array of promises and wanted to iterate over the values with JavaScript's forEach method. As we will see this is problematic, however a solution is readily available. It took me a while to understand what was going on, but hopefully you can skip the pain that I had as I will explain it all here.

Disclaimer: The code in this blog post was reduced to highlight the source of the problem. This is not the best way to implement this functionality in JavaScript. It is instead meant to make the problem and resolution clear.

A promise-less starting point

I started with an array of values. I performed some logic on this array combining the values into one and then returned the result.

function getMovieNames() {
  let displayToUser = '';
  let movieNames = [
    "Lord of the Rings",
    "Band of Brothers",
    "Interstellar"
  ];

  movieNames.forEach(movieName =>
    displayToUser += movieName + ' ');

  return displayToUser;
}

alert(getMovieNames());

This is simple enough: It alerts a combined string of movie names. No problems so far.

A promise filled update

For reasons external to the problem at hand, I had to take the data that I was receiving and fetch other data with it. Now instead of having an array of values we iterated over these values and retrieved asynchronous data. Can you guess what get's returned?

function getReleaseDates() {
  let displayToUser = '';
  let movieNames = [
    "Lord of the Rings",
    "Band of Brothers",
    "Interstellar"
  ];

  movieNames.forEach(movieName =>
    displayToUser += fetchReleaseDate(movieName));

  return displayToUser;
}

alert(getReleaseDates());

An empty string! Of course you can't know without seeing the implementation of fetchReleaseDate, but if we assume it is making an HTTP request or taking some similarly asynchronous action, then getReleaseDates is returned before the forEach callback is ever executed.

Await to the rescue

If we need to await for the fetchReleaseDate method, it seems like the await keyword should work for that. Right? In order to await for fetchReleaseDate the function it is in needs to be labeled as async. In this case that is the callback that we are giving the forEach method.

async function getReleaseDates() {
  let displayToUser = '';
  let movieNames = [
    "Lord of the Rings",
    "Band of Brothers",
    "Interstellar"
  ];

  movieNames.forEach(async movieName =>
    displayToUser += await fetchReleaseDate(movieName));

  return displayToUser;
}

alert(await getReleaseDates());

Not so fast! Even if we await for the fetchReleaseDate method, the forEach method does not await for our callback. And so the result is no different than before.

For of to the rescue

If the callback not being awaited on is the problem, the solution is to remove the callback. To do this let's change out the use of forEach with a for...of block as shown below.

async function getReleaseDates() {
  let displayToUser = '';
  let movieNames = [
    "Lord of the Rings",
    "Band of Brothers",
    "Interstellar"
  ];

  for (movieName of movieNames) {
    displayToUser += await fetchReleaseDate(movieName);
  }

  return displayToUser;
}

alert(await getReleaseDates());

Finally! It works again. While this may be obvious to some, others like me who are addicted to the forEach and map style of using arrays may not see this gotchya with using async / await along side callbacks.

Parting tip: If you ever declare a callback function as async in order to use the await keyword inside, ask yourself first: "Is the function I am calling going to await on my callback?" If the answer is no, then you will need to refactor your use of the callback function.

Check out my blog for more of my musings upon technology and various other topics.

Web Component as Good as Direct HTML for SEO?

megazear7 — Mon, 08 Jul 2019 19:24:35 +0000

I have been thinking about web components, light dom vs shadow dom, server side rendering, and SEO. The question that comes to mind is whether or not using web components as a way to encapsulate design and functionality with no asynchronous requests actually has worse SEO than having that data directly in the light dom. I.e Is either of these better or worse for SEO:

<my-component title="Example"
              description="Some more content"
              button-text="Do something">
</my-component>

<div class="my-component">
  <h4>Example</div>
  <p>Some more content</p>
  <button>Do something</button>
</div>

Assuming that the web component implementing <my-component> synchronously creates the same html as the second example, just inside of the shadow DOM?

Do we have any real evidence for the first option being not as good for SEO as the second option?

Or has the idea been to play conservative since we can't be sure that whatever service is crawling our page will actually run JavaScript, and therefor the useful content of the page would be hidden inside of attributes instead of in semantic HTML elements?

Google Calendar API Integration Made Easy

megazear7 — Mon, 08 Jul 2019 14:37:37 +0000

I recently struggled to integrate the Google Calendar API while trying to create an event listing for a website. While in retrospect most of it should have been easy but there were a few tricky configurations that were required. I wanted to share the steps I took to integrate a website with the Google Calendar API. We will use JWT authentication which is the most secure and flexible way to connect to a Google API. Keep in mind this type of connection is to connect to a Google that you manage, not a users Google account.

Enable the Calendar API

First we need to setup the Google API and gather the various connection information that we will need in order to connect to it. To begin simply enable the calendar API.

The first step is to go to the Google API Console and either select an existing project or create a new project.
Then select the "Enable APIS and Services" button
Type "Calendar" into the search box
Click on "Google Calendar API"
Click "Enable"

Setup JWT Access

Next we need to create a JSON Web Token, commonly referred to as "JWT". This will provide us with the information that we need to programmatically access various Google API's in a secure context.

Go to the Google API Console
Choose "Credentials" from the left hand navigation
Click the "Create Credentials" button
Choose "Service account key"
This will download a JSON file to your computer. We will refer to this as the JWT JSON file throughout this blog post.

Retrieve the Google Project Number

Now we need to retrieve the Google Project Number which will be required by the Google API.

Go to the Google API Consolec
In the top right of the screen click the three dot menu option next to your profile picture.
Click "Project Settings"
Copy the "Project Number" and paste it somewhere for safe keeping. We will refer to this as throughout this blog post in code examples

Setup the Google Calendar

Finally we need to configure the Google calendar and retrieve the ID. This was the tricky step as you have to give the JWT token access to you calendar as mentioned in steps 5-7 below.

Go to Google Calendar
On the left hand side hover over the calendar that you want to list events for and click the three dot menu icon
Click "Settings and sharing"
Scroll down and copy the "Calendar ID" and paste it somewhere for safe keeping. We will refer to this as throughout this blog post in code examples.
Open up the JWT JSON file and copy the client email.
Go to the "share with specific people" section of the calendar and click "Add people"
Paste the client email and submit the dialog

Setup Example Express Project

Now let's create a sample project using Node.JS with an Express server. Create a directory for your project and run the following commands:

npm init
npm install express --save
npm install googleapis --save

Create Your Express Server

Finally we need to write code that will start up an express server to listen to requests, contact the Google Calendar API with our connection details that we retrieved above, and then reply with the result. Note that the constants defined at the start of the file need updated based upon the corresponding values we retrieved earlier. The and values both come from the JWT JSON file.

/index.js

const express = require('express');
const { google } = require('googleapis');

const app = express();
const port = 3000;

const SCOPES = 'https://www.googleapis.com/auth/calendar.readonly';
const GOOGLE_PRIVATE_KEY="<private-key>"
const GOOGLE_CLIENT_EMAIL = "<client-email>"
const GOOGLE_PROJECT_NUMBER = "<project-number>"
const GOOGLE_CALENDAR_ID = "<calendar-id>"

app.get('/', (req, res) => {
  const jwtClient = new google.auth.JWT(
    GOOGLE_CLIENT_EMAIL,
    null,
    GOOGLE_PRIVATE_KEY,
    SCOPES
  );

  const calendar = google.calendar({
    version: 'v3',
    project: GOOGLE_PROJECT_NUMBER,
    auth: jwtClient
  });

  calendar.events.list({
    calendarId: GOOGLE_CALENDAR_ID,
    timeMin: (new Date()).toISOString(),
    maxResults: 10,
    singleEvents: true,
    orderBy: 'startTime',
  }, (error, result) => {
    if (error) {
      res.send(JSON.stringify({ error: error }));
    } else {
      if (result.data.items.length) {
        res.send(JSON.stringify({ events: result.data.items }));
      } else {
        res.send(JSON.stringify({ message: 'No upcoming events found.' }));
      }
    }
  });
});

app.listen(port, () => console.log(`Example app listening on port ${port}!`));

Now run your server with node index.js and go to localhost:3000 to see the result. Remember to move the private key and other constants into environment variables instead of committing them to source control. You should see the next 10 events that were added to the Google Calendar. This can be used to integrate a Google Calendar with an upcoming events feature on a website or could serve as a starting point for other features requiring a Google Calendar integration. Lastly the JWT authentication shown here can be used for a wide variety of Google APIs.

Considerations for CMS Integrations with JAMstack Sites

megazear7 — Fri, 28 Jun 2019 14:42:30 +0000

It is often assumed that static site generators have limitations that server side rendering or client side rendering is necessary to get beyond. Single page applications, progressive web apps, engaging and dynamic UI's, integrations with databases and content management systems; these cannot be used with a statically generated site can they? They certainly can. One of the names this technique goes by is JAMstack. While in my opinion a server sending prebuilt HTML to a browser should be the standard it has become necessary to give it a name as both the server side and browser side of web development has become cumbersome and overly complex.

However there are many things to consider when building a JAMstack website. How do you integrate content and databases? How do you enhance it with modern web development techniques such as SPA and PWA? We will take a look at some of these considerations and specifically how they apply to web content management.

Fine Tuned Build Hooks

If your website is large enough, rebuilding it might be a heavy burden on your content provider and databases. How often builds occur, the source and scope of the changes, and the number of requests to your content providers and database that it requires all need considered when designing the build process. Imagine you are building a news outlet that will have tens of thousands of pages and data is changing hundreds of times per day. In a circumstance like this it is important to only rebuild the part of the site that is affected by any given content or code change. We will look at some of the ways to fine tune build processes to only rebuild the pieces of the website that are necessary so that our JAMstack site can scale indefinitely.

When it isn't a Problem

First it is important to understand when this is not a problem. If your site has about a hundred pages and rebuilds from code and data changes are expected a few times per day, then this should only produce a few hundred page builds. If your content providers are capable of millions of reads per day, then this is multiple orders of magnitude away from being a problem. In addition it is often easier to scale the capabilities of data and content providers than it is to deal with restrictive build processes. That being said there are circumstances where a fine tuned build process is necessary.

Build Hooks

When deploying a JAMstack site you usually need to rebuild your site any time dependent data or content changes. To do this your hosting solution needs to have build hooks so that other systems can notify it when it needs to perform a rebuild. The content providers, databases, and code hosting solutions that you select in turn need to be capable of web hooks so that they can notify the hosting service that a rebuild is necessary. They also need to be capable of passing on detailed information so that the build process can identify exactly what needs rebuilt. Additionally, your selection of a static site generator will either limit or enable fine tuned build hooks and will affect how much of your site needs rebuilt on each content or data update.

Global Data

Websites are always going to have global data. This might be the name, metadata, and other information that appears in all or many pages. Another example of this would be a "latest posts" section that appears on every page of your site. Any time this data changes each page might need updated, and there is no way to know until you actually perform the build. One solution is to restructure your site features to not run into this issue, but that is of course limiting and we would rather find a solution that enables us to maintain the feature. There are two general approaches we can follow to solve this problem.

Controlled Changes

The first option is simply to limit the number of changes to global content. This tighter control allows you to maintain the same build process, keeping a simple build system with maximal end user performance. This might come in the form of limiting who has access to updating the content or limiting how often changes to it are integrated into the build. The benefit of this is simplicity and end user performance. However this limits how flexible you can be with global data and how quickly changes can be integrated into the site.

Async UI

The other option is loading this content asynchronously after the page load from an always up to date API. This means that the data can change as often as you would like and changes can be integrated into the end user experience the moment they are made. The limitation of this option is complexity and performance as it introduces client side rendering. The page and any dependent libraries must be loaded, requests for data made, and the UI updated. This is the design that nearly all web frameworks follow, and is the reason for much of the slowness of the modern web. However sometimes it is necessary in order to implement highly interactive and data intensive UI. In these scenarios using this asynchronous UI approach in a careful and selective manner can give content flexibility and more real time content updates.

User Data

The other data consideration with JAMstack sites is authenticated experiences. Do users have their own data? Should some content require authentication? Are some views user specific, where the content they see depends upon their profile? In these scenarios there are some additional considerations with regards to how these views should be delivered.

Prebuilt Views

You can still follow the JAMstack pattern of prebuilt HTML, but this time delivered only through an authenticated UI. This means that your otherwise static server needs to be capable of authorizing requests, increasing hosting complexity, but it also means that you maintain the performance benefits of prebuilt HTML. It is worth noting that this not a very common solution and many build process will either not support this or require a lot of customization.

Async UI

The other option for authenticated experiences is Async UI once again. This one is the standard for many websites these days and for good reason: maximum flexibility. You simply load in your authorized data after the page load and build your UI with a JavaScript library. This enters us back into the world of JavaScript based UI's and needs to be carefully considered so that these portions of your UI do not slow down the initial page load or over reach their welcome. Any UI that can be prebuilt should be prebuilt. User specific UI that needs updated based upon user data and interaction likely will need generated after the page load. However it is easy for this to creep into the rest of the UI, slowing down initial page load and rendering, and unnecessarily generating the UI only after the user needs it, instead of having it ready for them.

Continuous Integration

JAMstack sites are a perfect fit for continuous integration made simple. Your build process already produces a site that can be simply and statically delivered. We have already discussed the need for build hooks that force a rebuild of the site. This same process can be easily extended into version control. All that is needed if for your coding hosting system such as GitHub to use a web hook that triggers the build hook of the hosting system.

However not every code change requires a full rebuild of the site. Very often static site generators will have a source directory hierarchy that matches the end website. In this case if only one page is modified in the source code then it is possible that only that page needs rebuilt.

Single Page Applications

It is common to think that static site generation cannot be a single page application. This is in fact far from the truth. Combined the two can give you the best of both worlds, SPA architecture reducing the amount of network traffic and static site generation reducing the amount of run time processing. Single page applications increase performance by loading only the information needed to update the view, without reloading and re-rendering the whole page. Static sites are good at limiting the amount of run time processing, simply delivering prebuilt HTML as the user needs it. These can be combined to produce a website that is maximally performant.

To do this the static site generator that you select needs to be capable of rendering not only the full static page, with metadata, footer, header, and other layout elements included, but also needs to generate the HTML of just the page. This way the client code can load in these views upon navigation without loading in the whole page layout and metadata

Progressive Web Apps

Progressive web apps surely require a more complex infrastructure than what static site generators are capable of right? Again, this is not the case at all. In fact they go hand in glove. Service workers often require ahead of time understanding of the structure of your site and static site generation provides just this type of information that might otherwise not be known until runtime in a server side or client side rendering scheme. Progressive web apps can turn your static HTML and JavaScript into an installable, offline capable app that can access remote API's, real time databases, authentication services, and cloud functions that turn them into true apps with rich features.

JAMstack Suggestions

With these considerations in mind, I have the following suggestions when implementing a JAMstack site alongside a CMS:
Enable fine tuned build hooks my maintaining a meaningful site hierarchy and selecting the right content, hosting, and site generating solutions.
Enable single page applications by selecting a static site generator capable of generating content in various formats such as with layouts included and not included.

Enable user specific experiences with client side rendering solutions that can fulfill their purpose without overtaking the entire app.
Limit the amount of global data changes and places where small content changes would require full site rebuilds by planning your page layouts and features ahead of time. It is also important to know ahead of time which features will have such an impact so that you can plan accordingly.
Know which side of the build cost vs data request cost equation you are on. Are your databases capable of fulfilling orders of magnitude more requests than your build requires? Then skip the fine tuned build process. Will you have millions of product pages where rebuilding them even once is a heavy load on your database? Then carefully plan ahead for a fine tuned build process.

Check out my blog for more of my musings upon technology and various other topics.

Signs That Your Unit Tests Are Not Helpful

megazear7 — Mon, 10 Jun 2019 16:47:56 +0000

Here are two metrics that you can look at with your unit tests:

Percentage of test failures that are only a problem with the unit test, not a problem with the code running in a real environment. This often happens when you are mocking interfaces.
Percentage of times that a bug fix or otherwise non breaking change requires an update to a unit test.

The goal would be to have both these percentages at 0%. This would indicate that every test failure represents a real issue in your code and that only feature changes require updates to the unit tests.

As these percentages approach 100% the less helpful your unit tests become. However what percentage should act as the cut for when your unit tests are a net positive verse when they are a net negative? We can make two assumptions:

Every test failure that reflects a real code issue exactly out weighs every false positive.
Every unit test update that comes from a non feature change is out weighed by the benefits of having a unit test that comes from a feature change.

With these two assumptions in place then 50% for each of the metrics is the cut off for when your unit tests become a net negative. These are broad assumptions that might not always be true, but would need determined on a project by project basis based upon the technology that is in place. However it does provide a framework for thinking about the utility of unit testing.

Check out my blog for more of my musings upon technology and various other topics.

Pinterest, Facebook, and the Future of Social Media

megazear7 — Fri, 10 May 2019 10:28:52 +0000

The world wide web promised to allow anyone and everyone to distribute content to the world. In previous generations newspapers, magazines, radio, and television were the mediums of content distribution. This meant that the content we consumed was funneled through centralized authorities. The news, art, opinions, and stories that we read, heard, and watched had to come from one of these outlets.

The promise of the web was to break down this centralization and control by giving everyone access to distribute content. The centralized authorities that dominated content distribution of previous generations would come to an end and people would be free to connect and share with one another directly. Enter social media; the height of this promise of personal connection.

Users of social media platforms do not see themselves as content creators, but that is indeed what they are. They produce serialized stories and opinions, provide inspiration, and in many other ways produce content that others want to consume. Social media promised to deliver this freedom of the world wide web to everyone. However it was actually the downfall of the very thing it promised to deliver.

Facebook, Twitter, YouTube, name a dozen others; content distribution is more centralized now than ever. While we legally own the content that we upload to these sites, they are the ones with the power over who see's that content, how they see it, and what is allowed to be posted. The cultural and physiological affects of this content distribution is only recently being understood even as authority over content is once again in the hands of a few. We can share and interact with people all over the world like never before. However, those interactions are constantly manipulated by the platforms on which they occur. Worse still, the content that we produce is tied to the specific platform. You post to Facebook, Twitter, and YouTube. Legally you own the content, however it is inextricably tied to the platform.

Many have picked up on this growing back peddling of the promise of the web. Open source social media platforms allow algorithms to be inspected, new versions of the site to be spun up, and usually provide other sources of security that the primary social media options do not provide. These include Minds, Mastodon, Diaspora, and the like. Some of these have grown into millions of users and by being open source those users have visibility and insight into what that social media platform is doing. This also provides an easy way to replace that platform if necessary. However this content is still tied to the platform, and so I do not think this model is the full solution.

Back to the popular; Pinterest is not primarily a place where individuals upload content. Pinterest does have control of how that content is distributed, however it does not have direct control over the content itself. Rather, it links to content from other sources. The content itself can be hosted on any publicly accessible platform and then redistributed on Pinterest. This is the next piece of the solution: content should be individually hosted and then redistributed on any and every platform.

Individual content hosting means that we publish content to our own domain. Instead of our latest post, tweet, or video getting uploaded to one of the big social media companies, it is uploaded to one of a wide array of content services and then published to an individuals website. This gives individuals full control over publishing their own content without handing it over to the aforementioned centralized content authorities.

Once we have replaced centralized content publication with individual content publication we still need a way to connect and share. However now there could be many platforms that enable you to discover and share this individually published content. Preferably these would be open source platforms that can be inspected and monitored by the public and can be easily replaced if necessary. We could even use different content discovery platforms and still access one another's content because that content was individually published instead of being published to a specific social media platform.

With the tie between individual content and social media platforms being broken, the growing psychological affect that these platforms have on us as a culture can be mitigated, our content can be future focused as any new discovery platform can have access to all the same content, and ultimately it makes these technological powerhouses easily replaceable. Currently, to leave the major social media platforms means that you become disconnected from the content that people publish to that platform. Under this new model you could leave a content discovery platform if you deem it to be malicious or harmful and yet still remain connected to all the same people and content under a new platform.

However there are challenges. How do you convince a billion people that they are in fact content creators? How do convince them to publish their content differently? How do we make individual content publication as easy as centralized publication? How do we create new content discovery platforms that can reach individually published content all over the web in a way that can replace the existing social media platforms? This is a wide array of challenges and will require solutions from a wide array of domains such as social psychology, social outreach, and technology. However the cultural shift to social media took a decade, and we should expect any attempt to shift to a new and healthier model to take just as long.

Check out my blog for more of my musings upon technology and various other topics.