DEV Community: andrewingram

Deriving A Data Model From A Design With GraphQL

andrewingram — Thu, 25 Jun 2020 13:25:42 +0000

Data modelling is one of those skills that’s needed at nearly all levels of the software development stack. It covers the full spectrum from information architecture to UI, to APIs, all the way to the underlying database tables. When it’s done well, everyone benefits. Customers can easily form accurate mental models, designers and engineers can work together effectively, and analysts can extract and interpret data in a meaningful way. All of this makes it a useful skill to develop.

In this article, I introduce a fun little exercise I came up with; one that I sometimes use when interviewing candidates. The challenge is to pick a single screen or page from an app or website, and derive as much of its data model as you can. I like to pick popular web apps like Twitter and Reddit. If you're interviewing someone, and have a suitable example, you could even use your company's product. For this article, I'm using Dribbble.

Picking a page

The most suitable page to use is going to vary by product, for Twitter I've used the homepage for logged-in users, likewise for Reddit. For Dribbble, I’m going with a Shot page. I've picked one more-or-less at random after browsing the site a bit. Like most individual examples, it doesn't contain every possible element, but that's not a problem.

There’s more than one way to break down the problem, for this article I’ll tackle it top to bottom.

As a fan of GraphQL, I like to use its Schema Definition Language (SDL) to write down the data model. I've found the SDL is a great way to have conversations about data, so it's become my format of choice. In an interview situation, I wouldn't impose any constraints as to how someone writes down their solution, as long as I can understand it.

Breaking it down

Starting at the top

Above the main shot image, we can see a few pieces of content. The most obvious is the title of the shot. We can also see who created it (or posted it, if we're being picky), and who commissioned it, the latter of which I know to be optional. There are two avatar images, with the creator being the larger image, and the commissioner being the smaller of the two. There are also buttons for following the creator, saving the shot, and liking the shot. The state of these buttons changes depending on whether the viewing user has their respective actions already.

So we already have a few entities at play: the shot, the creator and the commissioner, as well as a few fields on each of them. For the sake of simplicity, I’ll assume the creator and commissioner are represented by the same type of entity, let’s call it a User.

type User {
    id: ID!
    name: String!
    avatar: Image!
    viewerHasFollowed: Boolean!
}

type Shot {
    id: ID!
    title: String!
    createdBy: User!
    commissionedBy: User
    viewerHasLiked: Boolean!
    viewerHasSaved: Boolean!
}

The viewer terminology in the above schema is a common GraphQL idiom that refers to the person “viewing” the data, usually the currently logged-in user. For brevity, I've decided not to define the Image type.

The main event

Let's move onto the focus of the page, the shot image itself. There's not a lot going on here, so it's fairly simple.

# I'm using the type extension syntax (http://spec.graphql.org/June2018/#sec-Object-Extensions) for brevity
extend type Shot {
    image: Image!
}

Done. Well, kind of. I know from being familiar with Dribbble that there’s a bit more to it than this (such as GIFs and videos, and allowing multiple images), but it adds complexity that’s wise to skip on the first pass of the exercise.

The left column

The first thing in the left column is the description of the shot; it supports some degree of formatting and there are lots of ways this could be represented in an API. Let’s assume there's already an HTML type we can use.

extend type Shot {
    description: HTML!
}

The next part is the comments. Aside from the list of comments itself, we have a total comment count, as well as the ability to display them with different orderings. Looking at a single comment, we can see a few pieces of information:

The comment text
The author name, avatar and “pro” status
The comment timestamp
Whether the “viewer” has liked it or not
The number of likes (not shown in the picture is also the fact that you can click the number of likes to see the individual “likes”)

Let’s build out the Comment type.

extend type User {
    isPro: Boolean!
}

type Comment {
    id: ID!
    author: User!
    content: HTML!
    shot: Shot! # For completeness, not strictly needed
    submittedAt: DateTime!
    numLikes: Int!
    viewerHasLiked: Boolean!
    likedBy: UserConnection!
}

I’ve implemented the likedBy field as a GraphQL Connection rather than just as an array to support pagination. My reasoning here is that even though the number of likes for a single comment will usually be quite low (compared to likes on a Shot itself), it still essentially has no upper bound. This means we can’t assume we’ll be able to efficiently load them all at once, thus making pagination a requirement.

The most common oversight I see with this exercise is to not consider performance, so some mention of pagination is key. A related issue is that I often see people wanting to let the front-end do the work of calculating aggregate values like total counts (numLikes is an example here). Both issues are usually seen together. If someone sees the need for pagination, they don't usually miss that it makes reliably calculating totals in the front-end impossible.

There’s potentially another missing capability here -- if the system wants to treat a "like" as a first-class entity, then we might replace the likedBy connection with a likes one referring to some kind of Like type rather than User. I haven’t seen anywhere on Dribbble where likes are used as anything other than a join between users and shots (e.g no additional data such as timestamps), so I've opted to keep things simple.

Now we have a way of modelling comments, we can add them to shots

enum CommentOrdering {
    OLDEST_FIRST
    NEWEST_FIRST
    MOST_LIKED_FIRST
}

extend type Shot {
    numComments: Int!
    comments(orderBy: CommentOrdering = OLDEST_FIRST): CommentConnection!
}

I've added an ordering parameter to the comments connection to support the three different choices we can see in the UI.

The right column

Nearly everything we can see here is related to the Shot itself, and there are quite a few new pieces of information:

Tags
Extracted colour palette
Number of likes (not visible is that it clicks through to a page showing a list of all the liking users)
Number of saves (also a link to a page showing which collections the shot has been saved in)
Timestamp posted
A selection of up to 4 other shots by the same creator

extend type Shot {
    tags: [Tag!]!
    colors: [Color!]!
    numLikes: Int!
    numSaves: Int!
    likedBy: UserConnection!
    savedIn: CollectionConnection!
    postedAt: DateTime!
}

extend type User {
    shots(exclude: ID): ShotConnection!
}

I haven’t defined Tag and Color here, but they could either just be simple scalar string values, or implemented as richer data structures. I've chosen not to define what Collections look like because they’re not part of this page. For the shots field on User, I've allowed an ID to be excluded so that we don’t show the shot we’re already looking at, but there are several ways we could solve this problem.

Wrapping it up

We’ve reached the bottom of the page and covered most pieces of data we can see. If I were giving this exercise in an interview, I wouldn’t expect a good candidate to get everything, and I certainly wouldn’t expect them to make the same assumptions and decisions as me. But I do think this is a valuable skill worth testing for, and this exercise is a lot more fun than asking people to solve an unfamiliar recursion problem on a whiteboard. If a candidate gets through it quickly, you can ask follow-up questions (performance ones are good), or click through to another page and see how the model can be refined or expanded.

This is just one of many ways to test for data modelling skills. For backend developers, there are other exercises you can use. However, when assessing front-end or API-oriented developers, this can be a great exercise to incorporate into your interview process.

Full solution as gist.

Demystifying GraphQL Connections

andrewingram — Thu, 28 May 2020 23:38:37 +0000

If you’ve used GraphQL for a while, it’s likely come across its (formally Relay’s) Connection Specification whether you’ve used it or not. It’s a pattern for implementing cursor-based pagination in GraphQL. Relay itself comes with first-class support for working with connections, but the pattern is often used in the wider GraphQL ecosystem — such as with Apollo.

Over the years I’ve been working with GraphQL, I’ve seen a number of misconceptions pop up time and again. I’ll quickly address each of them before we dive into a deeper explanation of connections.

Misconception 1: Connections are a pattern unique to GraphQL

As will hopefully become clearer by the end of this post, the problems that connections solve are not unique to GraphQL. Assuming performance and coherent data modelling are a concern, you’ll likely face these problems regardless of how you choose to implement your API. What’s different with GraphQL is simply that there’s a prominent first-party specification written by the original authors of the technology, and implemented in its earliest versions.

Misconception 2: Connections are the only way to implement pagination with GraphQL

Due to a relatively high degree of prominence within GraphQL’s own documentation, as well as the wider ecosystem, it’s easy to be left with the impression that you have to use connections if you want to implement pagination with GraphQL. Whilst connections are a good pattern and I’d advocate using them if possible, you’re ultimately free to implement pagination in any way you choose.

Misconception 3: You have to use connections if you’re using Relay

A commonly cited reason for not using Relay, is that you have to use connections (along with a number of other patterns). It’s true that Relay originally placed quite a few hard constraints on schema design, but nearly all of these were lifted with its 1.0 release (aka Relay Modern) in 2017. Interestingly, the use of connections was never a requirement — Relay merely provides a number of APIs and optimisations for working with them. That said, I’d argue that most (if not all) of the patterns promoted by Relay should be considered recommendations, regardless of whether you’re using it or not.

What problems do connections solve?

When discussing connections, it’s important to realise that they’re solving two problems:

The need for a coherent and complete data structure that lets us implement pagination for anything in our schema in a common way.
The need to implement pagination efficiently

The GraphQL Connection Specification solves both of these, by providing a well-considered data structure and promoting the use of cursors rather than the more common limit-offset pagination you might be used to seeing. It’s important to recognise that two problems are being solved, because then it should hopefully become clearer that even if (say) you don’t want to use cursor-based pagination, the overall shape of the connection specification is still worth adhering to.

One thing that’s important to bear in mind with connections, is that if you think too hard about their overlap with graph terminology, it’ll probably harm your understanding more than it helps. So when we discuss things like edges, nodes and connections, do your best to leave any existing graph knowledge from computer science at the door.

The structure of a connection

Let’s say you had an Event type you wanted to paginate over. If you were to use the connection data structure, you’d end up defining an EventConnection type as follows (using the GraphQL SDL):

type PageInfo {
    startCursor: String!
    endCursor: String!
    hasNextPage: Boolean!
    hasPreviousPage: Boolean!
}

type EventEdge {
    node: Event!
    cursor: String!
}

type EventConnection {
    edges: [EventEdge]
    pageInfo: PageInfo!
}

To use this type, you might do so like this:

type Query {
    upcomingEvents(
        first: Int
        last: Int
        before: String
        after: String
    ): EventConnection
}

The arguments to upcomingEvents and the structure of PageInfo itself are specific to cursor-based pagination, but I want to draw closer attention to EventEdge. The edge type is particularly contentious, I’ve often seen it argued that it’s unnecessary.

The primary reason why people think it’s unnecessary, is that all the information you need to actually perform pagination is on PageInfo, making the additional cursor field serve no purpose. And if the cursor field isn’t needed, can’t we omit the entire edge and just jump straight to the nodes?

Defending edges

Earlier I mentioned that connections solve two problems, and I have two equivalent defences of the inclusion of edges in the specification.

The role of cursors on edges

I’ll start with the justifications specific to cursor-based pagination. These are all about unlocking potential performance optimisations. I say potential because as far as I’m aware there’s currently no automatic behaviour in Relay (or anything else) that takes advantage of per-edge cursors, so the fact that they’re required by Relay might be considered a legacy wart. One simple use case is related to mutations. Should a mutation cause a previously fetched connection to change sufficiently for its pageInfo to no longer be valid (perhaps you deleted the last edge or otherwise moved it), having a cursor on every edge will allow you to synthesise an updated version of it without having to refetch, thereby fixing the state of the pagination data in the store and letting you continue to load more results.

A slightly more complicated involved use case for per-edge cursors is related to the fact that you could request connections with different page sizes. Let’s say for a given connection you’ve already queried the first 10 results, then elsewhere in UI you choose to query the same connection but this time you only want the first 5. In terms of the data you likely care about (the nodes), you don’t actually need to fire off a query because you already have all the data you need. The problem is the pageInfo you’ll have for the component rendering 10 results won’t be correct for the one rendering 5. But as with the mutation use case, if we have per-edge cursors, we have all the necessary information to synthesise a correct pageInfo without actually having to execute a query.

At this point I should clarify why we need the edge type at all, and absolutely shouldn’t be considering putting the cursor on the node itself. In my earlier example I had an Event type which you’d expect to have an id field on it. With GraphQL, it’s all but standardised that for a given query, anything with an id should look the same no matter how many times it appears or how it’s accessed (allowing for different field selections) — GraphQL clients such as Relay and Apollo depend on this rule to work correctly. So when you need to add transient contextual information to something, the correct pattern is to use a wrapper. In the case of connections, the wrapper is the edge type.

Other uses for edges

So we’ve established that there’s some value in having a cursor on each edge type. But what if you’re not using cursor-based pagination, or don’t care for those optimisations, do we still need an edge wrapping each node?

Imagine you’re building a mobile app that shows you places around a location. You have a UI that shows you some information about the place, and its distance from you, probably showing the closest first. You choose to model it as follows:

type LatLng {
    longitude: Float!
    latitude: Float!
}

input LatLngInput {
    longitude: Float!
    latitude: Float!
}

type Place {
    id: ID!
    name: String!
    location: LatLng!
    distance: Float!
}

type PageInfo {
    startCursor: String!
    endCursor: String!
    hasNextPage: Boolean!
    hasPreviousPage: Boolean!
}

type PlaceEdge {
    node: Place!
    cursor: String!
}

type PlaceConnection {
    edges: [PlaceEdge]
    pageInfo: PageInfo!
}

type Query {
    placesAroundLocation(
        location: LatLngInput!
        first: Int
        last: Int
        before: String
        after: String
    ): PlaceConnection
}

Perhaps you can already see the problem, if the user were to issue a query with different locations, the distances on each place should change — but this violates the aforementioned common caching assumption that a type with an ID is the same no matter how you get to it. If it’s possible to have the same place but with different values for its fields, something has gone wrong in our modelling.

So we change our modelling a bit:

type LatLng {
    longitude: Float!
    latitude: Float!
}

input LatLngInput {
    longitude: Float!
    latitude: Float!
}

type Place {
    id: ID!
    name: String!
    location: LatLng!
}

type PageInfo {
    startCursor: String!
    endCursor: String!
    hasNextPage: Boolean!
    hasPreviousPage: Boolean!
}

type PlaceDistanceEdge {
    node: Place!
    cursor: String!
    distance: Float!
}

type PlaceDistanceConnection {
    edges: [PlaceDistanceEdge]
    pageInfo: PageInfo!
}

type Query {
    placesAroundLocation(
        location: LatLngInput!
        first: Int
        last: Int
        before: String
        after: String
    ): PlaceDistanceConnection
}

We’ve made a few changes here. The most important is that the distance field has been moved out of the Place type and onto the connection edge. This is because it represents transient metadata about the node that’s only true for the current connection. This change also highlights that our connection and edge types aren’t universally re-usable, if we were to allow place connections where there’s no use for distance, we’d want to use different connection and edge types. As a consequence, I’ve renamed the connection types to highlight that they’re specific to this use case.

Note: there’s nothing stopping you having generic connection types with lots of nullable fields, but my preference is for more explicit modelling to lock down the possible shapes of data in client code

A similar use case might be a search feature where you want to display the relevance of each result to the search query, this information belongs on the edge because it’s contextual to the connection.

But do we always need edges?

So there are some valid use cases for edge types, be it for Relay’s cursors, or something more specific to your needs, but if none of these apply, do we still need edges on our connections?

The simple answer is no, but you might still choose to implement them. The reason is that we get some benefit from having common patterns. Even if most pagination use cases have no need for edges, the fact that some do means that it’s arguably advantageous to use them anyway, to add a degree of consistency (and to open up potential abstractions) in client code.

One compromise I’ve seen is allowing direct traversal from the connection to the nodes (skipping the edge type) for simpler scenarios, whilst retaining the edge type for when they’re genuinely needed, so the modelling looks like this:

type EventConnection {
    edges: [EventEdge]
    nodes: [Event]
    pageInfo: PageInfo!
}

I think this is a good pattern, I’ve yet to implement it myself but would advocate for it as a quality-of-life convenience.

Limit-offset pagination

I said at the start that most of the connection structure would be the same even if we weren’t using cursor-based pagination. Now that I’ve justified the inclusion of edge types, it’s possible to show how similarly we’d model what is arguably the most common kind of pagination in use today.

type LimitOffsetPageInfo {
    totalCount: Int!
}

type EventEdge {
    node: Event!
    cursor: String!
}

type EventLimitOffsetConnection {
    edges: [EventEdge]
    pageInfo: LimitOffsetPageInfo!
}

type Query {
    upcomingEvents(limit: Int!, offset: Int = 0): EventLimitOffsetConnection
}

You may notice the equivalent PageInfo type is quite sparse. You don’t actually need much information to construct the UI for this kind of pagination, but you may choose to add in additional data to simplify the work of building client UI — it really is up to you.

It’s worth noting that implementing cursor-based pagination rather than limit/offset is a decision that should be made mindfully. It’s true that it’s usually (perhaps always?) impossible to implement limit/offset efficiently because of the way that database queries get executed, but opting for cursor-based pagination ties your hands in terms of the kind of user interfaces you can build. There’s no magic bullet, and as with all technical decisions, you want to understand the choices you’re making.

I've seen some suggestion of a pattern where you combine both pagination schemes into a single data structure and allow the client to choose which arguments and fields to use. I can see the appeal, and definitely think there's a benefit to providing more choice if your backend can support it. My preference here would be to implement them as two independent parallel patterns rather than a combined one, if only because it communicates intent more clearly to consumers of the API.

In closing

So there we have it, hopefully it’s now a little clearer why the connection specification exists and what problems you can solve by adopting it. Perhaps you’ve also gained some additional insight as to where you can chop and change parts of it to suit your own individual requirements. Either way, thank you for reading!