DEV Community: Joseph Moore

Database Views Don't Really Exist

Joseph Moore — Thu, 09 Aug 2018 03:10:04 +0000

I keep running into a misconception about database views, even when talking to fairly experienced developers. People seem to be under the impression that, in a relational database, a view is something that actually exists. They don't!

Now, let me explain. Obviously, when you create a view, SOMETHING exists. But many developers seem to misunderstand what it is that exists. Most often, I encounter this misconception when trying to address performance problems - "Hey, if your join is so bad and slow, why don't you just create a view! That'll make your SQL faster, with no joins or anything!"

When I say that the view doesn't exist, I mean that, at any given time, there is no data stored that says "here's what's in this view!" A view is not a table that gets constantly updated when the underlying tables get updated.

So what is a view, really?

A view is a stored query. When you create a database view, the database stores the SQL you gave it. Then, when you come along and query that view, the database takes the stored view query, adds in the extras from the query against the view, and executes it. That's it! Essentially, a view is just an automatic query rewriter, letting you write SQL which is simpler than what you would otherwise write. It's kind of like writing a function to hide a bunch of reusable nasty code. That code still gets executed, it just gets hidden behind a simpler, reusable facade.

Examples, please!

Let's say you have two shiny tables, Cars and Drivers, which have a many-to-many relationship:

People can drive many cars, and one car can be driven by many people.

Now, let's say I need to be able to query for the name of everyone who drives a blue Toyota. What might my SQL look like?

select distinct(d.name) from car_drivers cd
left join car c on c.vin = cd.vin
left join driver d on d.driver_id = cd.driver_id
where c.color = 'blue' and c.make = 'toyota';

(This isn't the most performant version of this query, but we'll get to that in a minute.)

While this is a trivial example, it's still a lot of stuff to get right. In a real-world example, those joins can get out of hand quickly.

So, let's make a view, so our queries can be simpler!

create view cars_and_drivers_vw as
select c.vin, c.color, c.make, c.model, c.year,
       d.driver_id, d.name, d.birthday
from car_drivers cd
left join car c on c.vin = cd.vin
left join driver d on d.driver_id = cd.driver_id;

Now, my query becomes:

select distinct(name) from cars_and_drivers_vw
where color = 'blue' and make = 'toyota';

Nice! Simple! Pretty!

But what's really happening?

Here's what happens when we set all of this up in an Oracle database and ask Oracle to show it's execution plan for our simple query:

If you don't know what you're looking at, never fear! That's what I'm here for!

What this shows us is that Oracle is taking this:

select distinct(name) from cars_and_drivers_vw
where color = 'blue' and make = 'toyota';

and turning it into this:

select distinct(d.name) from car_drivers cd
left join car c on c.vin = cd.vin
left join driver d on d.driver_id = cd.driver_id
where c.color = 'blue' and c.make = 'toyota';

(Hey, that looks familiar!)

Nice!

Stop Telling Me Views are Faster!

Now we come to the thing that made me write this post. Views do not improve performance! In one of my team's more complex applications, we have some very large tables with complicated relationships and even more complicated queries that need to be performed on them. Every time we run into a slow query, someone suggests that we can solve the slowness with a view. By creating a view that does all of our joins for us, we can magically make our query faster! However, if you realize that a view is just a fancy query re-writer, not a persistent data structure, you know that this is not true. Using a view will not be any faster than the raw query; in fact, it could easily make things worse. The answer to performance problems is not a view; it is often a process of trial and error: finding different queries that yield the same results, analyzing query execution plans, and finding good indexes to help the database query things more efficiently.

As an example of a query that might be more efficient than the slow one we've been using, here's a query that is equivalent to our first query, but may be more efficient over a large data set.

select d.name from driver d
where exists (
    select 1 from car_drivers cd 
    where cd.driver_id = d.driver_id 
    and exists (
        select 1 from car c
        where c.vin = cd.vin 
        and c.color = 'blue' 
        and c.make = 'toyota'
    )
);

What about materialized views?

You just had to bring those up, didn't you.

Materialized Views are a concept which exists in Oracle and in Postgres (and some other, smaller engines). The term is really a misnomer, and Oracle has replaced the name 'Materialized View' with 'Snapshot', as it is a more accurate description of what is happening.

When you create a ~~materialized view~~ snapshot, the database runs the query you specified immediately, then stores the results. The important thing is that, as the original tables change, the contents of the materialized view do not change. You have to manually refresh the view whenever you want to update the contents, which can be a slow operation. So, materialized views are fast, but they rapidly become stale. They're mostly useful for non-real-time analytics and analysis, where you would periodically refresh the view, then perform lots of queries on it.

The exception to this (since there always is one in SQL databases) is Microsoft SQL Server, which allows you to create an 'indexed view,' which is like an Oracle snapshot, except that updates to the underlying table are reflected in the view. However, this comes with limitations to the queries which can be used to construct the view.

Epilogue

So, obviously, we didn't use a view to solve our recent performance problems. In fact, the initial version of the application used views for these queries, and we got rid of them because they actually made many of our queries much slower. We also can't solve our specific problem with indexes because the most complex parts of the queries are split across two tables 🤦.

Instead, we leveraged some mechanisms we'd already built into the app to implement a small bit of CQRS - we created a small table that keeps the important, hard-to-optimize parts of our common queries. We already had hooks in our DB layer that allowed us to run additional updates as side effects to the updates on our main tables, so we took advantage of that to make sure that our query table is always up-to-date. This allowed us to add some very nice indexes, and one of our queries went from taking about 20 seconds to taking about 0.15 seconds.

Bringing Order to Web Design Chaos (with Web Components)

Joseph Moore — Fri, 09 Feb 2018 05:43:39 +0000

Or, how I learned to stop worrying and love Web Components

In my maturation as a Software Engineer, I've found that stories of the problems and solutions that other engineers have encountered to be invaluable tools for improving and learning. So, I'm going to start sharing some of my code stories. This is the first of (hopefully) many, and it is not a short one. Cross-posted from my blog

This is the story of an interesting engineering effort I've participated in over the last year. It's been a really fun side project for me (~20 hours a week at the start, ~5 hours a week since then), which has allowed me to experiment with some fun new technologies and implement some cool stuff. It centers on an effort by a large, cross-disciplinary team to bring a simple, unified theme to all of our many, many web properties that can share code between any CMS or frontend framework.

Where We Started

I work for Brigham Young University, a large, private University.

Universities are a unique environment. In a university like ours, you have a number of large colleges and schools, all of which have a lot of freedom to do what they want. Historically, one way in which this freedom has manifested itself is in Web design, which has lead to some very...inconsistent designs.

For example, this time last year, a designer I work with looked at the home pages of all of our top-level colleges and schools, and found that they all had completely different designs. They used different logos, layouts, shades of our school colors (navy blue and white), and, in one extreme case, didn't use our school colors at all. Many of these pages were loosely based on a design that had been published in 2013. This particular design was never well documented and the reference implementation wasn't portable across platforms, so we saw a lot of divergence and changes as time went on.

I don't mean to shame any particular schools, because most of their designs actually looked quite nice, but here's a ~~horrifying~~ fun picture to show what I'm talking about:

After hiring an outside firm to conduct an audit of our web presence, it was decided that it was time to try and bring some brand uniformity to the University. An initiative was launched in our web community to design and implement a simple, modern, unobtrusive standard theme and to encourage developers and designers across campus to converge on this theme.

I'm not going to talk about the design part, because I'm not a designer. While I was asked to attend the design meetings, it was as an engineer who would be asked to develop a reference implementation of the theme. I should note, however, that it was decided to only require a small header (including search and user sign-in), simple navigation, and a footer. Everything else was up to individual sites to design, though we've also been working on providing them with off-the-shelf styles and components so that every site doesn't have to redesign common elements.

Our Goals

Once our design group hammered out a design that everyone could work with, it was time for our engineering group to implement things. While the design group worked, we had one department that stepped up in a big way and developed a Drupal module that allowed them to rapidly iterate on the design across several of their sites, so the designers could get feedback on what worked in the real world and what didn't. However, for long-term use, we decided that we wanted a solution that gave us three things:

[ ] Cross-platform
[ ] Simple
[ ] Fixable

Cross-Platform

We have a wide variety of web hosting platforms across campus, ranging from simple static sites, to CMSes like Drupal and Wordpress, to complex single-page apps written in Angular, React, and other frameworks. We needed a solution that would allow a small team to make our theme work consistently across all platforms. Ideally, this would mean having one single codebase for everything.

Simple

We have a wide variety of skill sets on campus, ranging from seasoned, 30-year veterans to CS students who got a part-time job maintaining a departmental site. We needed a solution that was simple and easy to consume, even if the developer doesn't understand everything that's going on - if you can't reliably copy and paste from examples, it would be too complicated.

Fixable

One of the problems we had run into in the past was that, as the theme was improved, only new sites would adopt the improvements. Because every site kept their own copy of the theme code, there was no good way to distribute updates and fixes.

Web Components to the Rescue

A few of us in the group had been playing with Web Components, and the v1 version of the specifications had just been implemented in Chrome. After looking at several proposals on how to reach our goals, we decided that Web Components would give us a lot of what we were looking for.

Web Components solved a lot of our problems. Because they're a platform-level primitive, we didn't have to worry as much about integration - they can (theoretically) integrate with anything that can work with the DOM.

To put this in perspective, it took a couple of student programmers who hadn't seen our code before about 30 minutes to implement a Wordpress theme that was based on our Web Components. My favorite demonstration of the ease with which we could integrate came when I was doing a presentation about the theme to the campus organization I work for. As a live demo, I took one of our really old, nasty pages and retrofitted it to use the new theme, right there in front of everyone. It took about 10 minutes. Granted, I had practiced this beforehand, but it was still pretty fun to show just how easy it could be.

Beyond the ability to write them out to the DOM, Shadow DOM gave us some really nice style isolation, so we could be certain that none of the site's styles could accidentally mess up our themed components, and none of our internal styles would leak out to the outside world.

There have been some hiccups on the way, but our CMSes have had a pretty easy integration story, and we've got Single-Page Apps in production that are using React (16+), Angular (2+), VueJS, and others. There are a handful of SaaS tools doing some truly awful things to the DOM that we haven't gotten working yet, but for the most part, it's been pretty uneventful.

Goal Check In:

[x] Cross-platform
[ ] Simple
[ ] Fixable

Web Components also promised to solve some of our complexity problems. If you're familiar with Bootstrap, you know how, in order to use their styles, you have to structure your markup in just the right way, with lots of nested divs and special classes. It's really easy to get lost in all of that structure, especially since the design we were given had some very large layout differences between mobile and desktop views. The concept of slots in Shadow DOM gave us a really easy way to hide the markup complexity from the consumer, while still allowing them to plug in all of their site-specific things.

(If you're unfamiliar with Shadow DOM, slots are a lot like doing component composition with children in React or <ng-content> in Angular)

Our final header markup looks something like this:

~~(Note that the fonts aren't the actual fonts, as those can only be loaded on *.byu.edu sites. Open the full codepen to see the desktop view.)~~ This isn't true anymore - we swapped the fonts, and I didn't have to change the codepen at all! #winning

At first glance, there's a lot going on here. However, most of the structural complexity is hidden, and only the things that are specific to a given site are included.

To give you an idea of how much structure has been left out, here's what this looks like after being rendered by the Shady DOM polyfill, which combines all of the content into one DOM tree:

<byu-header mobile-max-width="1023px" max-width="1200px" class="byu-component-rendered" left-align="">
    <div id="header" class="byu-header-root">
        <div class="byu-header-content needs-width-setting stretches">
            <div class="byu-header-primary">
                <a class="byu-logo" id="home-url" name="home-url" href="https://byu.edu/">
                    <img class="byu-logo" alt="BYU" src="https://cdn.byu.edu/shared-icons/latest/logos/monogram-white.svg">
                </a>
                <div class="byu-header-title">
                    <a slot="site-title" href="some-site.byu.edu" class="">Site Title</a>
                </div>
            </div>
            <div id="secondary" class="byu-header-secondary">
                <div class="byu-header-user">
                    <byu-user-info slot="user" class="byu-component-rendered" has-user="" role="button">
                        <div class="byu-user-wrapper">
                            <div class="no-user slot-wrapper">
                                <div class="user-info-image" aria-label="User Icon"> </div>
                                <span class="text slot-wrapper">
                                    <a slot="login" href="#login">Sign In</a>
                                </span>
                            </div>
                            <div class="has-user">
                                <span class="name slot-wrapper">
                                    <span slot="user-name">Joe Student</span>
                                </span>
                                <div class="user-info-image" aria-label="User Icon"> </div>
                                <span class="logout slot-wrapper">
                                    <a slot="logout" href="#logout">Sign Out</a>
                                </span>
                            </div>
                        </div>
                    </byu-user-info>
                </div>
                <div class="byu-header-search">
                    <byu-search slot="search" action="submit-form" class="byu-component-rendered">
                        <div id="search-form" class="">
                            <div id="search-container" class="">
                                <form action="search.php">
                                    <input name="search" class="__byu-search-selected-input" placeholder="Search" title="Search" type="search">
                                </form>
                            </div>
                            <button id="search-button" type="submit" class="">
                                <img id="search-icon" src="https://cdn.byu.edu/shared-icons/latest/fontawesome/search-white.svg"
                                    alt="Run Search" class=""> </button>
                        </div>
                    </byu-search>
                </div>
            </div>
        </div>
    </div>
    <div class="menu-outer-wrapper">
        <div class="menu-inner-wrapper slot-wrapper needs-width-setting" style="max-width: 1200px;">
            <byu-menu slot="nav" class="byu-component-rendered">
                <nav class="outer-nav slot-container needs-width-setting">
                    <a href="#link1">Link 1</a>
                    <a href="#link2">Link 2</a>
                    <a href="#link3">Link 3</a>
                    <a href="#link4">Link 4</a>
                </nav>
            </byu-menu>
        </div>
    </div>
</byu-header>

I've actually simplified that a little. It's not pretty, and Web Components give us this nice little wrapper behind which we can hide all of this complexity.

Goal Check In:

[x] Cross-platform
[x] Simple
[ ] Fixable

Dynamic Distribution

(This is the part I'm really proud of)

Once we had a working set of components, we needed some way to distribute them to developers on campus. We considered using some sort of package manager, like NPM, but we quickly determined that it would not fit our needs. While it would give us a simple way to push out updates, there would be no way to guarantee that sites would see the update. We have sites that will go years without seeing an update, and, in the event of an important bug, we wanted to be able to have even those sites get the update. In addition, with the variety of platforms we needed to support, it would be hard to converge on a single package manager that would be easy for everyone to use.

The solution we converged on was to build a central CDN and have all sites load resources from it. This has turned out to be a great idea. We built a process that could respond to webhook notifications from Github and push code changes to the CDN. My office has been aggressively pursuing a move to Amazon Web Services, so we build the CDN using Amazon S3, Cloudfront, and CodeBuild.

Our CDN's secret sauce is its assembler. When we make any pushes to any of our Github repositories, an AWS Lambda function receives a webhook notification and kicks off a build running on AWS CodeBuild. We use CodeBuild because a) we don't have to run any servers for it and b) it allows us to execute basically anything that can run in a docker container. The assembler looks at a list of repositories that are included in the CDN, then uses the Github API to find all of the branches and tags that have changed or been created since the last assembler run. It pulls down the necessary files, then copies the required files into our S3 bucket.

In order to provide automatic updates, we include the concept of 'aliases' in the CDN. Aliases use semver parsing of Git tags to create aliases like latest (always the highest version number), 1.x.x (gets all minor and bugfix releases), and 1.1.x (gets all bugfix releases for the 1.1 major release). We also include an unstable alias, which points to whatever is currently on master. Users can also stick to a specific release by providing an explicit version number. This allows us to provide automatic updates to all most sites, while giving more control to those who desire it.

When a consumer wants to use something from our CDN, they construct a URL like this:

https://cdn.byu.edu/{library name}/{version or alias}/{file name}

So, the URL for the main Javascript for the always up-to-date version our theme components is https://cdn.byu.edu/byu-theme-components/latest/byu-theme-components.min.js. Getting the current latest version and turning off auto-updates would have a URL of https://cdn.byu.edu/byu-theme-components/1.2.3/byu-theme-components.min.js.

We want to maximize the cacheability of all of our assets, but this can complicate the rapid rollout of new releases. We've found a good compromise by assuming that tags will never change (which is usually true). So, we serve anything coming from a specific tag with cache headers that indicate that it is cacheable forever (Cache-Control: immutable, public, max-age=31557600, s-maxage=31557600). Then, when a user requests an alias (like latest), we send back a 302 redirect to the specific tag and set a cache expiry on the redirect to 1 hour. This gives us a good balance between having updates roll out often and keeping useful things in the user's cache. The redirects happen quickly enough and transfer little enough data to the user that they don't add much overhead, and we keep the vast majority of our assets cached in the browser forever (well, a year in most browsers, but that might as well be forever, right? I mean, it's longer than the lifetime of most Javascript frameworks).

Goal Check In:

[x] Cross-platform
[x] Simple
[x] Fixable

Additional Challenges

There are always problems that you don't foresee going into a project (or even some that you foresee, but decide not to tackle until later). Here are some of the ones we faced.

Opting-out of a Framework

When we started working on this a year ago, Polymer 2 was still a few months from being released, and we weren't enamoured with Polymer's insistence on using the never-going-to-be-implemented HTML Imports specification. SkateJS was an option, but we didn't really like the way Skate was handling templates.

Faced with these options, we decided early on that we didn't really need a Web Component framework. Our components were going to be fairly static - stamp the DOM once, allow for a few small dynamic things, and be done with it. So, we thought, it wouldn't be that terrible to use the raw Custom Element APIs, augmented with a small collection of helper functions for doing common things (like stamping a template to the Shadow DOM).

If we were starting this project now, I think we would be using some form of framework, probably ~~Polymer 3 (even though it's not released yet) or SkateJS 5 (with its nice new pluggable renderers)~~ LitElement. While our setup isn't THAT complicated, we do still re-invent the wheel in a lot of ways.

// Detect dark theme var iframe = document.getElementById('tweet-904902463588839424-826'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=904902463588839424&theme=dark" }

Singing the Load-Time Blues

That still left the problem of loading our elements. With the HTML Imports spec, you define your elements in an HTML file, which contains your markup, your styles, and your JS code, all in once nice happy file. You pop a <link rel="import"> into your page, and your components magically load everything they need.

However, we have a lot of skittish developers on campus, and we didn't want to introduce many new concepts to them (Yes, there are some who would get scared off by something as simple as a new type of <link> tag). We wanted to be able to stick with simple <script> tags for loading. You add into that the fact that nobody but Chrome wanted to implement HTML Imports, and things were looking pretty dismal for the Polymer 1 and 2 way of loading things. (Sidenote: I still long for a better solution to importing HTML than just 'stick it all in Javascript').

We also wanted our development experience to be nice. If we couldn't just bring in an HTML document with embedded CSS and JS, we wanted to be able to keep our CSS, JS, and HTML in separate files, but still have them all be imported with one simple script tag.

First, we wrote a simple Gulp module that would assemble things for us, but it was finnicky and relied on comments that were formatted in juuust the right way. That's when we discovered Webpack, which solved this problem for us. Now, we could have arbitrary dependency graphs, use modules from NPM, and just generally get things done without having to write our own bundler. That has worked really well for us and, as a result, importing our theme component bundle is simple:

<script async src="https://cdn.byu.edu/byu-theme-components/latest/byu-theme-components.min.js"></script>

Styling the Unstyleable

Shadow DOM is great for keeping our template CSS contained, but what about styling the pieces that the consumer hands us in our slots? The whole point of this project is the styling!

Shadow DOM does provide an answer for styling slotted content, but it's a limited one. I can make use of the ::slotted pseudo-element to style them:

<style>
  .my-slot-wrapper ::slotted(*) {
    color: navy;
  }
</style>
<div class="my-slot-wrapper">
    <slot></slot>
</div>

You'll notice that ::slotted takes in another selector. That selector can be any CSS selector, with one limitation: it can only be one level deep. You can't have a selector like ::slotted(div > a), it can only be one level deep.

Most of the time, this limitation isn't a problem, but we quickly ran into two major issues: styling links and styling inputs.

As soon as we published a beta version of our theme components, we saw people adding links to their site titles that were nested inside of <h1>:

<byu-header>
  <h1 slot="site-title"><a href="my-site.byu.edu">My Site</a></h1>
</byu-header>

Normally, nested text elements wouldn't cause any problems for us - we set color, font-size, and font-family on the parent element, and it cascades down. Links, however, are special. Every user-agent stylesheet I've seen applies their own color and text-decoration to links, and doesn't inherit those styles from the parent. If the <a> is the element with the slot attribute, this isn't a problem - the styles get directly applied to it. However, when the link is nested inside another element, we can't style it from within the Shadow DOM.

The other problem we ran into was styling search inputs. We were aiming for compatibility across a wide range of frameworks and CMSes, and each one has their own ways of wrapping inputs that we couldn't get rid of. However, we needed to make sure that their inputs got styled properly. One solution could have been to present our own styled input and use Javascript to keep the two up-to-date with each other, but we felt that this approach had a lot of potential to confuse developers. For example, it would make implementing something like an autocomplete popup very, very difficult.

So, we needed to be able to have this:

<byu-search>
  <div class="my-cms-wrapper">
    <input type="search" />
  </div>
</byu-search>

get styled just like:

<byu-search>
  <input type="search" />
</byu-search>

The solution to both of these problems turned out to be something we were already familiar with: external CSS stylesheets. We added a stylesheet (we called it the 'extra' styles) containing special selectors to handle these and other corner cases, and we have developers include both the stylesheet and the Javascript in their pages:

<link rel="stylesheet" href="https://cdn.byu.edu/byu-theme-components/latest/byu-theme-components.min.css">
<script async src="https://cdn.byu.edu/byu-theme-components/latest/byu-theme-components.min.js"></script>

Developers were already used to including pairs of CSS and Javascript files (like with jQuery UI), so we weren't introducing anything new to them.

Because we were using custom element names, the selectors in the 'extra' stylesheet can be very simple, yet specific:

byu-header > [slot="site-title"] a {
    text-decoration: none;
    color: white;
}

While we try to avoid selectors that are more than two layers deep, having the outer two layers (byu-header > [slot="site-title"]) gives us pretty good scoping, so our styles don't bleed over and affect the styles on the page. There's still the possibility that some stylesheet later on the page will do this something like

a {
    color: green;
}

We decided that, since we can't solve everything, we won't try to solve this problem. After all, this is just an instance of CSS behaving as expected, so developers should be able to figure out why the site title is suddenly green all on their own (well, with the help of their browser's devtools, at least).

<blink> Considered Harmful (Flash of Unstyled Content)

One of the pitfalls to using JS to render parts of a UI is the Flash of Unstyled Content. This is a common problem across frontend frameworks and isn't specific to Web Components, though I hope that proposals like Declarative Shadow DOM can help reduce the impact of it when using Web Components.

Typically, the strategy for overcoming this is to embed some CSS in the initial HTML that will only apply until the framework has loaded and starts rendering things. We don't have a good way to have people embed styles and still preserve our goal of easily pushing out updates, so we've compromised and embed the Flash of Unstyled Content styles in our 'extra' styles. This CSS file is much smaller and parses much quicker than our JS, so we can get a very good experience from it. Once the components have stamped their template, they apply the byu-component-rendered class to themselves, so all we need to do in our styles is look for components that don't have the class:

byu-header:not(.byu-component-rendered) {
    /* nasty fallback styles here */
}

Since our CSS gets resolved synchronously, this allows our pages to look almost-right while we wait for the JS to finish its work. This leads to practically seamless page transitions, especially once the browser has cached the CSS.

Future Work

One thing we want to add in the future is client-side analytics. While we can gather a fair amount of information from our server logs, we can't see some critical things, like how many sites are using different configuration options and component combinations. We're looking at using navigator.sendBeacon to send light-weight analytics back to our servers with information like the build of the code that has been loaded, which components actually get used, different features of each component that are being used, etc. We don't want to rely on an external analytics site, because we don't want to add the concerns about load time and privacy that come with them to all of our sites. If individual sites want to use a full-featured analytics tool, they'll have that option without anything we do interfering with that.

Another thing we want to add is automated tests. There aren't a lot of logical tests we can do, since there isn't a lot of logic in our components, but we are really looking forward to adding tests that use visual diffing tools like Pixelmatch to detect when we've made changes that alter the styling of our elements. The goal will be to have a step in our CDN assembler that runs the tests and doesn't deploy the new code if they fail.

Summary

This has been a really fun project to work on, forcing us to tackle a lot of interesting challenges. I'm really proud of the solutions we've come up with, as well as the way that a group of developers from different departments and of different skill levels all came together to come up with a solution that works well for everyone. While our code isn't perfect, and we haven't really had a chance to bring it all in line with a single style, I feel like we've done a pretty good job.

For those who would like to take a look at our code, it is all part of the BYU Web Github Organization and is licensed with Apache 2.0.

A few quick links:

Our main theme components
- A gallery of the theme components
Our CDN source
Drupal and Wordpress CMS themes that leverage the Web components.

Emojifying my Bash Prompt (and why you should too)

Joseph Moore — Sat, 29 Jul 2017 02:11:45 +0000

Warning: Dumb Stuff ahead

This post contains some Dumb Stuff. If you take yourself too seriously, you may want to go find something a little less whimsical to read ðŸ˜.

/warning

Let's face it - development work can sometimes be hard on the psyche. When the unrealistic project deadlines approaching or the mountains of technical debt are looming, sometimes a small little pop of happiness can get you through the day. One of the things that keeps me sane is finding little ways to give myself small bursts of joy throughout the day. That's why, when I got a new laptop a few weeks ago, I decided to play around for a bit and find a fun way to solve some problems. How, do you ask? By emojifying my bash prompt.

Why, oh why, would you do such a dumb thing?

Well, it all started with two different problems I'd noticed on my old laptop; minor little annoyances that were never worth taking the time to solve when everything else was working. I have a personal rule that, once I get a new laptop, I a few days to tweak and configure it, and then my setup is frozen for at least a few months. This helps me incrementally improve my setup, but prevents me from falling into an endless cycle of tweaking and testing and never getting any actual work done. I've seen developers get so lost in configuring their workstation that they're basically useless for a month, and I never want to be That Guy.

A Very Fancy ðŸŽ© Prompt

I've always had a thing for customizing my bash prompt; I never like the default. My last few workstations and all of my servers have had basically the same prompt:

hostname:/current/directory/here
 ->

This is, obviously, nothing special, but customizing my prompt was one of the first things I figured out how to do in Bash, and it still gives me a little burst of joy to remember the feeling of triumph when I finally got it to work how I wanted.

A couple of laptops later, I further improved this prompt to detect when I was in a subdirectory of my home directory, and replace my home path with ~. Yet another little burst of joy when I see it.

And now, I was setting up a new laptop, and I'd gotten bored with my prompt. First, I thought about adding colors to it, but I quickly got bored with that - it wasn't any more exciting than a plain prompt. Then, while installing some packages using brew, I noticed the little beer emoji that pops up when you successfully install something, and I got to thinking - what if I played with Emoji? Lately, whenever I need to come up with a trivial demo or example, I lean towards doing something dumb with emoji, because it makes me laugh to treat something as dumb as emoji seriously (just look at the output of a yeoman generator I created a while back to demonstrate some concepts to my team).

So, how to make my prompt Fancy?

Well, here's how:

I really didn't do much - I just replaced my ~ with a ðŸ and my little ASCII arrow with a â©. And yet, those little emoji make my day a bit brighter each time I see them.

You'll notice that I added another emoji - on this laptop, I'm keeping all of my work-related code under ~/work, and so I decided to replace that path with an office building ðŸ¢. Why? Well, why not? I might eventually add more such replacements, but I've already exhausted my self-enforced tweaking time. Perhaps I'll make ~/Documents be ðŸ“ and /tmp â³. I can definitely think of a few project directories that could be replaced by ðŸ’©. But that's a problem for another laptop.

So, my prompt setup isn't too complicated at this point:


fancyprompt() {
  echo

  where=$PWD
  home=$HOME
  work="$home/work"

  where="${where/$work/ðŸ¢ }"
  where="${where/$home/ðŸ  }"

  PS1='$where
â©  '
}

PROMPT_COMMAND=fancyprompt

But can it get fancier?

Of course!

Keeping me up-to-date

Once again, while installing all of my normal brew packages, I got to thinking about one of the problems I run into with Homebrew: outdated packages. I'll let so many updates accumulate without ever doing anything about them that, when I do need to update one, I end up with a brew upgrade running for half an hour, and I don't like that. So, could I find a fun way to solve this problem? Well, would I be here telling you about it if I hadn't? I don't think so!

The idea I settled on was to enhance my already-fancy prompt my making it tell me when there were brew updates to install:

My initial implementation looked like this:


fancyprompt() {

  brew update
  outdated=`brew outdated --quiet | wc -l | tr -d '[:space:']`

  if [ "$outdated" -gt "0" ]; then
          echo -e "\x1B[104m\x1B[97mðŸº You have $outdated brew update(s).ðŸº  \x1B[0m"
  fi

  ...the rest of my prompt...
}

This worked...slowly. brew update is very slow, as it has to do a lot of work, including a fair amount of network IO. So, I moved the brew update into a cron job that ran daily:

PATH=/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/sbin:/usr/local/bin

@daily brew update

(Who uses cron on their workstation? I do, apparently!)

Now, I could remove the brew update from my prompt. My prompt was then...less slow. As it turns out, brew outdated was pretty slow too, taking about 500ms to execute. Knowing exactly nothing about what's actually happening, I choose to cast the need to fire up a Ruby process to run brew as my red herring. Regardless, 500ms is way. too. slow. It felt like I was dying every time I typed a command and waited for my prompt to come back. So I had to get crazier.

I solved the speed problem by moving the brew outdated check to my cron job too, writing the output to a little file in /tmp:

@daily brew update && brew outdated --quiet | wc -l | tr -d '[:space:'] > /tmp/.brew-outdated

Now, I had a file that contained a count of my outdated brew packages. All I had to do was read it in my prompt function, which is very quick!

    outdated=0
    if [ -f "/tmp/.brew-outdated" ]; then
      outdated=`cat /tmp/.brew-outdated`
    fi

Hooray! Now, I get to happy little beer emoji and a nice message whenever I have brew updates.

But can we go fancier?

Never fear, dear reader, we can!

Update ALL THE PACKAGES

Turns out, I have a bunch more package managers to keep up to date. I do some web dev, so of course, I have a bunch of npm packages, and I have to have pip installed, because a bunch of our deployment tooling is in Python. So, can I extend this to get updates for those?

But of course!

I ended up creating a few extra functions to handle this for me, and wrote a script to check for updates:


#!/bin/bash

check_brew() {
  echo checking brew
  brew update
  brew outdated --quiet | wc -l | tr -d '[:space:]' > /tmp/.brew-outdated
}

check_npm() {
  echo checking npm
  npm outdated --global | wc -l | tr -d '[:space:]' > /tmp/.npm-outdated
}

check_pip() {
  echo checking pip2
  pip2 list --outdated --format=legacy | wc -l | tr -d '[:space:]' > /tmp/.pip2-outdated
  echo checking pip3
  pip3 list --outdated --format=legacy | wc -l | tr -d '[:space:]' > /tmp/.pip3-outdated
}

check_brew
check_npm
check_pip

Each of these writes to a file named /tmp/.(toolname)-outdated. I then have some functions in my bash profile that, given a tool name and an emoji, will check the appropriate file and show a message.


_count_updates() {
  type=$1
  f=/tmp/.$type-outdated
  upd=0
  if [ -f "$f" ]; then
    upd=`cat $f`
  fi
  echo $upd
}

_has_updates() {
  type=$1

  cnt=`_count_updates $type`
  if [ "$cnt" -gt "0" ]; then
          return 0
  fi
  return 1
}

_update_prompt() {
  type=$1
  icon=$2

  upd=`_count_updates $type`
  if [ "$upd" -gt 0 ]; then
    echo -e "\x1B[104m\x1B[97m$icon You have $upd $1 update(s).$icon \x1B[0m"
  fi
}

# configure my multi-line prompt
fancyprompt() {
  echo

  _update_prompt brew ðŸº
  _update_prompt npm ðŸ“¦
  _update_prompt pip2 ðŸ
  _update_prompt pip3 ðŸ

  where=$PWD
  home=$HOME
  work="$home/work"

  where="${where/$work/ðŸ¢ }"
  where="${where/$home/ðŸ  }"

  PS1='$where
â©  '
}

I then decided that remembering the right commands to update everything was a pain, mostly because Pip is weird, so I created a function in my bash profile that knows how to do it for me:


update_all() {
  if _has_updates brew; then
          echo "Updating Brew"
          brew upgrade
  fi

  if _has_updates npm; then
          echo "Updating NPM"
          npm update -g
  fi

  if _has_updates pip2; then
          echo "Updating Pip2"
          pip2 list --outdated --format=freeze | cut -d = -f 1 | xargs -n1 pip2 install -U
          echo "Running pip update again, because that's necessary"
          pip2 list --outdated --format=freeze | cut -d = -f 1 | xargs -n1 pip2 install -U
  fi

  if _has_updates pip3; then
          echo "Updating Pip3"
          pip3 list --outdated --format=freeze | cut -d = -f 1 | xargs -n1 pip3 install -U
          echo "Running pip update again, because that's necessary"
          pip3 list --outdated --format=freeze | cut -d = -f 1 | xargs -n1 pip3 install -U
  fi

 echo "Finshed updating"
 ~/bin/fancy-prompt/outdated-packages.sh
}

I ended up having to run pip2 and pip3 run their updates twice, because it almost always would end up with more packages that needed updating.

FANCIER!

No. I spent about an hour a day for a week tweaking this, and I've exhausted my setup time. In the future, I could enhance this to make it easier to add more package managers, or to swap them out and make it run on Linux. That, however, is a job for my next laptop.

Next Steps

If you want, you can download my scripts from (Github)[https://github.com/ThatJoeMoore/fancy-prompt]. I don't promise any kind of help or support, but I might make future tweaks.

I also suggest that you take a look at bash-git-prompt. It makes your git repos delightful ðŸ˜. I've even developed an emojified theme for it, because of course I did.

Conclusions, and maybe some Deep Thoughts

I realize that this is a completely ridiculous first post here, but I do actually have a reason for writing such a ridiculous post.

Do you regret spending so much time on something so trivial?

Nope. Even after almost a month, seeing these little emoji makes me happy. Perhaps the shine will wear off eventually, but until then, this was totally worth it.

But WHY!?!

Because it makes me happy.

So now we get to the deep part of my post.

Being a developer can be stressful. Often, there's a lot riding on us, and it usually doesn't come with much glory or recognition beyond our immediate team members. You hand a client a beautiful program on a silver platter, and they complain that it's not platinum and diamond-encrusted. We work late hours, we lose sleep to keep apps running, we consume far too much pizza (stop trying to make sure that my project team can still be fed with two pizzas!).

We all need to find ways to take care of ourselves, physically, mentally, and emotionally. Some may take their lunch break to exercise. Some may get happiness from having a nice standing desk, or a collection of Lego Star Wars sets, or by having pictures of their kids on their desk. For me, it involves emojifying my bash prompt.

If you just want to be a developer, sure, go ahead, skip the little things. But I think most of us also want to be a Human Being, and that involves more than code. If my experience shows me anything, it's that the best code comes from those who take the time to be a well-rounded person, and who find ways to make themselves happy throughout the day. To mangle a line from Mary Poppins, "A spoonful of sugar helps the developer stay sane."

And that's why, sometimes, I spend too long playing with something ridiculous, like emojifying my prompt. I do more than that to keep myself happy, but these little things can help keep the happy going throughout the occasional long day. Find the dumb things that work for you, and do them.

Hi, I'm Joseph Moore

Joseph Moore — Thu, 23 Feb 2017 20:12:36 +0000

I have been coding for 10 years.

You can find me on Twitter as @ThatJoeMoore

I live in Provo, Utah.

I work for Brigham Young University

I mostly program in these languages: Java and Javascript (browser and node).

I am currently learning more about functional and reactive programming. I dream of the day when I can convince my team to use a functional language like Clojure or Haskell.

Nice to meet you.