DEV Community: Andy Richardson

Packages don't solve distributed systems

Andy Richardson — Tue, 08 Feb 2022 11:11:34 +0000

Packages and libraries can be a great way to share functionality across different codebases. React, lodash, express - all of these are used across many codebases without issue.

So when it comes time to build out many backend services which operate within a similar domain - packaging logic can seem like a no brainier.

But what if I told you that's a disaster waiting to happen?

The scaling problem 📈

As companies grow, the business domain typically grows in complexity and size. Scaling this becomes a real challenge - both from a technical and organizational perspective.

As problem solvers, we break things down into smaller parts; typically, the business domain is broken into smaller "subdomains" - each with their own teams and services (Conway's law in full effect).

In an ideal world, each of those teams could work in complete isolation, doing what they do best and only having to solve problems within their domain.

In reality, an overlap of the domains of a business is common and significant overhead is introduced due to a need for cross-domain communication.

The same problem applies to technical architecture; while ideally services would be bound by the constraints of their own existence (and not of other services), domain overlap makes this hard to achieve.

Using packages 🎁

At face value, this seems like a very simple problem to solve. "I’ll make a package which abstracts the business logic into on place so all services can use it” said every web developer, ever.

While this seems like a no-brainer, what you end up with is a distributed monolith.

The efforts to solve scaling problems by breaking the domain down into services is undone when we move business logic into packages. We end up with a distributed system - albeit, one where services are coupled.

Coupling 👫

Coupling is the deadly sin of a distributed system. If the goal is to work at maximum velocity without the overhead of cross-communication, coupling achieves the exact opposite.

From the start of our careers as developers, we’re taught that code redundancy is bad, and that abstracting code into functions is good. While this is sometimes the case, it’s easy to get wrong, and if there’s anything worse than redundant code, it’s that one overly-abstracted function that does a milliion different things and is used across the whole system.

When we put business logic into packages, we end up with this exact problem... but on a completely new level.

Runtime coupling

The most obvious issue with distributing domain logic via packages is that, while the business logic might be nice and neatly placed in one place during development time, that won’t be the case at runtime.

Business logic needs to be consistent - so by distributing business logic via a package, how do you ensure consistency? Versioning won’t solve this issue because differing services could be using different versions of a package at any one time.

The solution I’ve seen over and over again is a monolithic release process; leading to slow CI times, a need for coordination amongst many teams, and volitile releases (due to transactional rollouts being outright impossible on a distributed system).

Code coupling

Problems for sharing business logic in packages spread further than just runtime cohesion. As services are tied together by the ones and zeros they use to operate, this can lead to restrictions on what those services can do and how they can operate.

Supported runtimes and dependencies of a package often fail to be updated as nobody is willing to take the risk to update to newer versions; lest it should cause the tightly knit tower of services to crumble.

This can often be a bidirectional problem whereby a service needs to update a package to address a bug, but that would in turn require changes to a shared package (and subsequently, many services).

Ownership 🙋

A less obvious consequence of using packages for business logic is the blurring of lines of ownership. In an architecture where services work within a bounded context (more on this later), all aspects are self contained - logging, metrics, reads/writes etc.

The same cannot be said when business logic from a package is running on any number of services at one time. Something as simple as “monitoring the creation of users” becomes a behemoth effort involving many services.

Before you know it, someone has built a company-wide logging library, further worsening the service coupling problem.

In the case of runtime issues, it is often unclear where the problem is located - the package, or the service. I’ve personally seen this lead to a culture of avoidant engineers who - understandably - don’t want to be accountable for breakages because there is no clarity on who is responsible for what.

Making it work 🤔

So by now, you likely understand why having many services share code can have a spreading impact on the stability, velocity and effectiveness of a distributed system and the teams behind it. What are our options?

Service Oriented Architecture 🏗

While the term itself sounds advanced, it simply refers to building a distributed system using services as the primitive building blocks.

If a distributed system is a jigsaw, services are the pieces which together create the full picture.

Bounded contexts 🔲

A pragmatic solution to coupling, bounded contexts are intended to create distinct boundaries between the different domains of a system.

Following the single responsibility principle, and coined by Eric Evans in Domain Driven Design, the goal is to ensure that any given service has a clearly scoped responsibility.

From a technical perspective, you can think of this as every running service being solely responsible for its existence, deployment, data and anything else relating to its domain.

By having a single source of truth for the different domains within a distributed system, many of the challenges mentioned earlier cease to exist.

Inevitable coupling 🚨

While having clear distinct boundaries sounds nice, in practice, things are never that straight forward. Services are always going to have overlapping concerns - and thats okay. We can solve that logical coupling in a manner that introduces minimal runtime coupling and no code coupling.

The internet

Okay so hear me out - there’s a way to communicate between machines using some wires and electricity - it’s going to take of soon!

GraphQL, REST, SOAP, you name it, any one of these is going to do just fine; and while you might have a better time with some protocols over others, as long as the only point of coupling between services is a (preferably versioned) protocol for communicating, code coupling is out of the question.

It can still go wrong

Excessive runtime coupling is still a possibility; and depending on how services communicate, a messy web of inter-dependent services is not impossible.

That’s a whole other topic in itself - I’d recommend this talk by Jimmy Boggard.

Ownership 🙋

Finally there’s the ownership angle. Those clear boundaries assigned to each service - assign them to people too.

Accountability isn’t just about knowing who broke that thing, it’s about giving teams the ability to have a say in what they do, and take pride in following that through to the end.

Its easy to say “not my problem” when some code that has been touched by 10+ different people breaks, but you’ll be hard pressed to find an engineer who is willing to respond like that when they proudly wrote all those bugs with their bare hands.

Better yet, you’re more likely to see engineers be proactive in maintaining their work when they know they’re responsible for the state of it.

Conclusion

So there you have it - hopefully this helps at least one team from ending up with a distributed monolith!

This is just the tip of the iceberg, so if you're interested in finding out more, I'd recommend the following resources:

Building Microservices (or basically anything else by Sam Newman)
You're not actually building micro-services
Domain driven design
Avoiding Microservice Megadisasters

Thanks for reading!

If you enjoyed this post, be sure to react 🦄 or drop a comment below with any thoughts 🤔.

You can also hit me up on twitter - @andyrichardsonn

Disclaimer: All thoughts and opinions expressed in this article are my own.

GraphQL requests over HTTP/s are slow

Andy Richardson — Mon, 02 Aug 2021 13:00:51 +0000

HTTP/s?

What year do you think this is - 2020?

It's 2021, and I'm here to tell you about a way you can make your GraphQL requests faster and more reliable - using WebSockets! 🚀

Traditional transport mechanisms

Most people familiar with GraphQL are accustomed to using HTTP/s for query and mutation operations; and there's good reason for that! HTTP requests are easy to manage and scale thanks to their simple call-and-response nature.

WebSockets on the other hand, while often more performant, require the management of many sustained connections. For "atomic" operations like queries and mutations, the runtime complexity and infrastructure costs introduced from using a long running socket has traditionally been an understandable deterrent.

What if I were to tell you there's a way to have the best of both worlds?

Managed sockets

As serverless technology has continued to evolve, stateless solutions have been introduced to address the otherwise stateful task of managing sockets.

In the case of AWS, Big Jeff's managed WebSockets was designed to solve this problem. At the time or writing, one million "connection minutes" costs a measly $0.25.

That isn't to say this is a perfect solution. The switch to a SAAS (sockets as a service) model involves requires a shift to a completely new API for working with sockets; and plug-and-play solutions are only just beginning to come about.

Improved performance (in theory)

Before considering a switch from HTTP/s to WebSocket, it's best to run through why this might be worthwhile.

With some exception, every query or mutation made over HTTP/s requires a new connection to be established - and that isn't free.

Opening and closing a connection requires some overhead which can introduce latency, thus making GraphQL requests take longer.

By instead using a WebSocket to communicate with a GraphQL endpoint, the single connection is sustained for the lifetime of the client - therefore removing the per-request overhead found with HTTP/s.

You can think of it like this:
HTTP/s: 100 queries/mutations -> 100 connections
WebSocket: 100 queries/mutations -> 1 connection

For lower level details, check out this stack overflow discussion

Performance isn't the only advantage. WebSocket connections typically have better fault tolerance, meaning clients on unstable connections should have an easier time sending and receiving messages.

Testing the theory

While the theory makes sense, I wanted to see if a measurable difference can be seen when making requests over a single sustained connection.

In order to get a result that measures the real-world impact + feasibility, rather than just the protocol overhead alone, I created an end-to-end project and benchmarked both protocols.

Performance results

Results when using a 5g connection

After testing this a number of times, I was relieved to see that there is a consistent performance improvement. But with the mere ~100ms difference in latency, I was a little disappointed.

Realizing that this is still a roughly 30% improvement in speed, I decided to explore whether the latency reduction would be more evident on slower connections

Results when using a simulated 3g connection

At this point, the impact became much more evident! With little to no effort or additional cost, I was able to measure an over half-a-second (~600ms) improvement.

Making the switch

So your GraphQL endpoint is already on serverless infrastructure, and you want to take the leap - what needs to be done?

If you're already using GraphQL subscriptions (on serverless infrastructure) for push-based data, first off - give yourself a pat on the back you trend setter 👏! There's no work required other than configuring your client to send requests via the socket rather than HTTP/s.

The likelihood however is that your endpoint isn't using GraphQL subscriptions. In the case of AWS, the serverless socket offering has been around for a few years but the work required to integrate this into existing sub-protocols has been fairly substantial.

I've been working to change this and created Subscriptionless which is a library designed to make socket-based GraphQL (queries, mutations, and subscriptions) simpler to implement on AWS's serverless infra.

If you want to take the leap, check out the repo for a guide and example project. You can also try out the end-to-end project repo which was used to do this performance comparison.

Conclusion

So there you have it - free network performance improvements at little to no cost!

Do we even need HTTP/s for GraphQL?

Do you see yourself using WebSockets more?

Share your thoughts below 💬

Thanks for reading!

If you enjoyed this post, be sure to react 🦄 or drop a comment below with any thoughts 🤔.

You can also hit me up on twitter - @andyrichardsonn

Disclaimer: All thoughts and opinions expressed in this article are my own.

GraphQL pagination with DynamoDB - Putting it together

Andy Richardson — Wed, 28 Apr 2021 10:15:07 +0000

Assuming you have a good understanding of relay pagination and DynamoDB pagination, here's a rundown on how to get the two working in harmony 🥂.

🐙 Creating a resolver

For the majority of this section, it's fair to assume that we're working inside of a resolver such as the following.

const usersResolver = () => async (
  root,
  { first, after, last, before },
) => {
  // ...
};

Determining direction

Before querying the database, we first need to know which direction is being requested by the user.

const isForward = Boolean(first);

The easiest way to do this is to see if the provided first argument has a value. If so, we're working with forward pagination.

Note: Bi-directional pagination within a single query is allowed but is not advised.

Querying the Database

For the query, most of the arguments passed through are going to be your bog-standard query; but there will be some additional attributes we need to pass through.

ScanIndexForward needs to be passed a boolean dependent on the direction of the query (i.e. isForward from the previous example).

ExclusiveStartKey is going to be the client provided cursor (i.e. before or after arguments). The current SDK doesn't support the value null so make sure to fall back to undefined for cases where a cursor isn't present.

await documentClient
  .query({
    ScanIndexForward: isForward,
    ExclusiveStartKey: before || after || undefined,
    // The rest of your query
  })
  .promise();

Page sizing

A single query won't be enough to guarantee that the client provided page size is satisfied. In order to work around this, we're going to need to create a utility to iterate through one or more DynamoDB pages to populate our collection.

export const paginateQuery = <R>(client: DynamoDB.DocumentClient) => async <T = R>({
  params,
  pageSize,
  acc = [],
}: {
  params: DynamoDB.DocumentClient.QueryInput;
  pageSize: number;
  acc?: T[];
}): Promise<{ page: T[]; hasNextPage: boolean }> => {
  const remaining = pageSize - acc.length;
  const result = await client.query(params).promise();
  const newItems = result.Items || [];
  const newAcc = [...acc, ...(newItems.slice(0, remaining) as T[])];

  // Query exhausted
  if (!result.LastEvaluatedKey) {
    return {
      page: newAcc,
      hasNextPage: newItems.length > remaining,
    };
  }

  if (
    // Page needs to be filled more
    newAcc.length < pageSize ||
    // page full but hasNextPage unknown
    newItems.length <= remaining
  ) {
    return paginateQuery(client)({
      params: {
        ...params,
        ExclusiveStartKey: result.LastEvaluatedKey,
      },
      pageSize,
      acc: newAcc,
    });
  }

  return {
    page: newAcc,
    hasNextPage: true,
  };
};

Having pieced this together, the previous DynamoDB query can now instead call this utility and the requested page size is passed along.

const { page, hasNextPage } = await paginateQuery(documentClient)({
  pageSize: first || last,
  params: {
    ScanIndexForward: isForward,
    ExclusiveStartKey: before || after || undefined,
    // The rest of your query
  },
});

Constructing Edges

The response from DynamoDB equates to the nodes in our response Edges. A cursor is also required to be colocated with these nodes.

In this example, the query is on a table (rather than an index) so the keys required correspond to the partition key and sort key of the table.

For index queries, see the Cursor Construction section of the DynamoDB Pagination post.

const cursorKeys = ['id', 'dateOfBirth'] as const;
const edges =  page.map((node) => ({
  node,
  cursor: cursorKeys.reduce((agg, key) => ({ ...agg, [key]: node[key] }), {}),
}));

Correcting edge order

While DynamoDB inverts sort-order when paginating backward, Relay does not. For this reason, the order of edges needs to be reversed if backward pagination is being used.

if (!isForward) {
  edges.reverse();
}

Constructing PageInfo

The task is almost complete! The final part to this pagination saga is to put together the PageInfo response.

Cursors

Assuming the edges have already been ordered correctly (see above) the start and end cursors can easily be set by getting the cursor values of the first and last edges.

const pageInfo = {
  startCursor: edges[0]?.cursor,
  endCursor: edges[edges.length - 1]?.cursor,
  // ...
};

Assuming the client is stateful, there's little-to-no need to tell the client whether there is a page available in the opposite direction. For this purpose, we can default to false for hasPreviousPage and hasNextPage for forward and backward pagination, respectively.

const pageInfo = {
  // ...
  ...(isForward
    ? { hasNextPage, hasPreviousPage: false }
    : { hasNextPage: false, hasPreviousPage: hasNextPage }),
};

Final result

Here's what our resolver looks like after putting all these parts together.

const usersResolver = () => async (root, { first, after, last, before }) => {
  const isForward = Boolean(first);
  const { page, hasNextPage } = await paginateQuery(documentClient)({
    pageSize: first || last,
    params: {
      ScanIndexForward: isForward,
      ExclusiveStartKey: before || after || undefined,
      // ...
    },
  });

  const cursorKeys = ["id", "dateOfBirth"] as const;
  const edges = page.map((node) => ({
    node,
    cursor: cursorKeys.reduce((agg, key) => ({ ...agg, [key]: node[key] }), {}),
  }));

  if (!isForward) {
    edges.reverse();
  }

  const pageInfo = {
    startCursor: edges[0]?.cursor,
    endCursor: edges[edges.length - 1]?.cursor,
    ...(isForward
      ? { hasNextPage, hasPreviousPage: false }
      : { hasNextPage: false, hasPreviousPage: hasNextPage }),
  };

  return { edges, pageInfo };
};

🚀 Conclusion

If you've gotten this far - congratulations! You are now a pagination expert™ and are ready to take this on in the real world 🌍!

For the sake of keeping this concise, I've left out a few additional steps (namely optimising cursors and input validation). If you would like to see that follow up post, be sure to let me know 💬.

Thanks for reading!

If you enjoyed this post, be sure to react 🦄 or drop a comment below with any thoughts 🤔.

You can also hit me up on twitter - @andyrichardsonn

Disclaimer: All thoughts and opinions expressed in this article are my own.

GraphQL pagination with DynamoDB - DynamoDB pagination

Andy Richardson — Wed, 28 Apr 2021 10:14:13 +0000

Just like the aforementioned GraphQL pagination, DynamoDB also uses cursor based pagination.

That being said, there are distinct differences that need to be taken into account 🔔.

🏎️ Traversal

For any given query to DynamoDB, the following takes place:

In-order traversal takes place on the target table (order specified by the sort key)
The provided key condition expression is evaluated to find matching documents
Optionally provided filter expressions are used to additionally constrain the matching documents

Inverting traversal order

Traversal order can be inverted using the ScanIndexForward attribute.

This (logically) inverted collection is traversed in reverse order, and the resulting items are also returned in reverse order.

📚 Paging

As matches are found, DynamoDB adds them to a result set - a "page". If a Limit is provided, DynamoDB will suspend traversal when the number of matches for the given key condition expression reaches the limit.

Unfortunately, the limit is applied prior to the filter expression evaluation; meaning a result set will never exceed the limit value, but can have a size smaller than the limit, while also having subsequent pages to follow.

🔑 Cursors

Upon the return of a page, assuming the collection hasn't been exhausted, DynamoDB provides a cursor in the form of a LastEvaluatedKey.

Providing this value in subsequent queries via the ExclusiveStartKey allows DynamoDB to continue where it left off.

Exhausting a query

In a similar fashion, this pattern can be used to retrieve all items in a query.

const exhaustQuery = async <T>(
  query: AWS.DynamoDB.DocumentClient.QueryInput,
  agg: T[] = [],
): Promise<T[]> => {
  const res = await dynamodb.query(query).promise();
  const newAgg = [...agg, ...(res.Items || [])];

  if (!res.LastEvaluatedKey || !res.Items) {
    return newAgg;
  }

  return exhaustQuery(
    {
      ...query,
      ExclusiveStartKey: res.LastEvaluatedKey,
    },
    newAgg,
  );
};

Cursor construction

DynamoDB responses only provide a cursor for the position of the last evaluated element in the collection. There doesn't look to be official documentation on how cursors are constructed, but I've found the following thanks to some experimentation.

It looks like DynamoDB needs two things to be able to continue where it left off:

The unique identity of the last visited item (i.e. primary key)
The position in the index where said element exists (i.e. primary key of index/table)

In the case of querying a table, the primary key typically consists of a partition (hash) and optionally a sort (range) key.

{
  productId: 1,  // partition key (pk)
  type: 'Book'   // sort key (sk)
}

For querying an index, the same rule applies. We'll still need the attributes required to uniquely identify the element (primary key), but we also need the partition and (optional) sort attributes to get back to the previous position in the query.

{
  productId: 1,          // pk
  type: 'Book',          // sk, index pk
  createdAt: 1618496921, // index sk
}

Thanks for reading!

If you enjoyed this post, be sure to react 🦄 or drop a comment below with any thoughts 🤔.

You can also hit me up on twitter - @andyrichardsonn

Disclaimer: All thoughts and opinions expressed in this article are my own.

GraphQL pagination with DynamoDB - Cursor specification

Andy Richardson — Wed, 28 Apr 2021 10:13:43 +0000

Note: This is a multi-part series; if you already know about relay pagination, feel free to skip to the next section.

Relay's Cursor Connection Specification has become the industry standard for pagination in GraphQL.

Whether you use a Relay, Apollo, or urql, the spec is supported by many client side libraries. This makes it an easy choice for those who don't want to implement their own client-side pagination logic (which is most of us).

💬 Pagination grammar

Before we declare a pageable field, you're going to want to know about the following primitives.

Edge

An Edge is a container type which ensures that each item in a paginated response (node) is accompanied with a position in the collection (cursor).

type UserEdge {
  node: User!
  cursor: String!
}

PageInfo

PageInfo is a metadata type which provides information about a pagination request.

type PageInfo {
  startCursor: String
  endCursor: String
  hasNextPage: Boolean!
  hasPreviousPage: Boolean!
}

Connection

A Connection is a container type which aggregates Edge and PageInfo values and is the root type of any paginated response.

type UserConnection {
  edges: [UserEdge!]!
  pageInfo: PageInfo!
}

🎨 Creating a pageable field

Adding pagination support to a field is pretty straightforward. The field needs to take the following four arguments - first, after, last, and before (more on this later) - and return a Connection.

type Query {
  users(
    first: Int,
    after: String,
    last: Int, 
    before: String
  ): UserConnection!
}

🏎️ Page traversal

Paginated fields take four arguments which can be divided into two distinct groups.

Forward pagination

Forward pagination is the process of fetching pages in a collection from top-to-bottom.

Arguments

The arguments to use forward pagination are:

first - the number of items to return (required)
after - a cursor resembling the position in the collection to start from (optional)

Retrieval

As you can see above, the first paginated query will include only the first argument. This tells the backend to start at the top of the collection.

For subsequent queries the after argument is used to instruct the backend where to continue from. This value corresponds to the cursor of the last edge on the previous response.

Backward pagination

Backward pagination is the process of fetching pages in a collection from bottom-to-top.

Arguments

The arguments to use backward pagination are:

last - the number of items to return (required)
before - a cursor resembling the position in the collection to start from (optional)

Retrieval

This second example might look familiar - that's because it's almost identical to forward pagination.

Just like with forward pagination, the page size is sent in isolation for the first query (this time last rather than first).

Again, for subsequent queries, a cursor is provided to instruct the backend where to continue from (this time before rather than after). This value corresponds to the cursor on the first (as opposed to last) edge on the previous response.

It's important to note, the sort order in which edges are returned is the same regardless of whether forward or backward pagination is used.

Thanks for reading!

If you enjoyed this post, be sure to react 🦄 or drop a comment below with any thoughts 🤔.

You can also hit me up on twitter - @andyrichardsonn

Disclaimer: All thoughts and opinions expressed in this article are my own.

How I exploited NPM downloads... and why you shouldn't trust them

Andy Richardson — Tue, 09 Mar 2021 10:46:38 +0000

Over this past month, I've managed to get a package, with little to no users, to accumulate over one million downloads 🚀.

It didn't cost any money, no laws were broken (I think) and it took little to no effort.

Here's what you need to know about the downloads statistic on NPM.

🔮 The illusion of downloads

If you've ever looked at using a new package from NPM, the chances are you've considered the "Weekly Downloads" statistic.

It's the first metric displayed on the page - so it must be useful information for the user... right?

A third of people who responded to this poll seemed to think so and even go as far as to say it has a large influence in their decision to adopt a new package.

But here's the thing, it isn't a useful metric for the following two reasons:

there is a loose (at best) relationship between users and download counts
the system is easily exploitable

What is a download

This was pretty well discussed on the NPM Blog but to summarise, it's any successful download of a package (tarball) from NPMs registry.

NPM have openly stated that this this statistic has no consideration for the source (IP, user agent, etc). Meaning all downloads are equal, whether it be from:

A user adding a new package to their project
A CI run installing dependencies
A bot downloading the package repeatedly to create the illusion of popularity (there's some foreshadowing for you)

As you can imagine, this means a project with frequent CI runs is likely to have more of an influence on download statistics than any set of individuals (especially when taking npm client caching into consideration).

Registries

The abundance of registries is another reason why download counts aren't an accurate reflection of usage. NPM download counts only include downloads to the official NPM registry, and not registries such as unpkg and github.

🧑‍💻 Exploiting the system

Disclaimer: I've documented this to bring light to how easily exploitable download statistics are. However, I strongly advise that you don't do this as it is both dishonest an unnecessary drain on NPM Inc's resources.

If you've read everything up until this point, you'll know that there isn't a need for any kind of "genius hacker exploit".

Instead, all we need is some way of downloading a package many times.

Running a script locally with some kind of cron job should do just fine - but that isn't too exciting... let's use serverless!

You can check out the full repo here.

Creating a script

For the Lambda, I created a function which takes the following arguments:

package - the package to download
probability - the likelihood of a download for a given run

The latter argument is intended to add noise - simulating the variable nature of downloads over time.

A "coin flip" takes place each run, with the probability argument being used to weight the chance of success. If the flip is successful, the package is downloaded.



export const handler = async ({ package, probability }) => {
  // Simulate coin flip
  if (Math.random() > probability) {
    // Flip fail
    return;
  }

  // Flip success
  await downloadPackage({ package });
};

Triggering the Lambda

To get this script running routinely, a CloudWatch event was set up that triggers at a rate of once a minute.



// Terraform example
resource "aws_cloudwatch_event_rule" "lambda_trigger_rule" {
  name = "trigger-npm-install"
  description = "Trigger an NPM install"
  schedule_expression = "rate(1 minute)"
}

Example CloudWatch Event Rule in Terraform.

In order to do something when this event is triggered, an event target is set up, pointing to the Lambda with our required arguments.



resource "aws_cloudwatch_event_target" "lambda" {
  arn = aws_lambda_function.install_package_lambda.arn
  rule = aws_cloudwatch_event_rule.lambda_trigger_rule.name
  input = jsonencode({
    package = "is-introspection-query"
    probability = 0.8
  })
}

Example CloudWatch Event Target in Terraform.

🚀 The result

After deploying this for the duration of a week, the result is... well actually not that impressive; it turns out there aren't as many seconds in a week as I had expected 🤔.

But alas, after some tweaks, we hit just under 1 million downloads per week!

Yes thats right, a package with literally 0 users has more downloads than the likes of urql and mobx.

Are you seeing the problem now?

Download stats don't work

Here's the thing, naive download statistics are useless at best and misleading at worst.

The large graph on NPM's site, the culture of celebrating download counts online, the third party sites which show package download "trends". These all contribute to this narrative that NPM download counts provide some kind of insight into a packages popularity, and they just don't.

Even ignoring the potential for malicious actors (like myself) the abundance of registries and caching implementations make these statistics less than useful.

"Popularity"

Fortunately, NPM has a saving grace - the popularity statistic! Let's just replace the download count with some of the more useful statistics... right?

Well no - turns out the popularity statistic seems to be the downloads statistic in disguise. As you can see below, my package managed to surpass @prisma/engines in terms of popularity.

Here's a quick comparison of the two packages side-by-side.

	@prisma/engines	is-introspection-query
weekly downloads	~100,000	~800,000
stars	264	0
forks	35	0
contributors	26	1
users	probably not 0	definitely 0

Conclusion

If there's one thing you take away from this discussion, it's that downloads alone aren't a useful metric.

While I've no doubt that NPM could create a popularity metric that aggregates a number of different attributes of a package (npms.io has already done it), from now on, I'm going to do a little more background research before trusting the downloads and popularity metrics on NPM 🕵️.

Hopefully, you found this interesting! If you have any thoughts or comments, feel free to drop them below or hit me up on twitter - @andyrichardsonn

Disclaimer: All thoughts and opinions expressed in this article are my own.

Whats new in Fielder V2

Andy Richardson — Fri, 01 Jan 2021 12:34:39 +0000

A new major release of Fielder is now live 🚀🎉

This latest release has been the result of countless hours exploring form design - here's what's new!

How we got here

With the initial release of Fielder, the goal was to create a form library that was unbound from existing form library constraints.

Removing the rigidity of monolithic validation schemas, Fielder posed as a simpler solution for forms by coupling validation to fields rather than forms.

const [fieldProps, fieldMeta] = useField({
  name: "username",
  validate: useCallback((value) => {
    if (!value) {
      throw Error("Username is required!");
    }
  }, []),
});

Example field-first validation in Fielder V1

This change in approach has proven to be effective in creating simple and flexible forms which can evolve over time (with exception).

This latest release compliments this mantra, continuing the focus on evolutionary and adaptive form design.

New features

Along with bug fixes and optimizations, this release comes with two major features which are designed to:

Make validation even more flexible
Remove the need for user-created form state

Validation events

Validation events are a new addition to Fielder which simplify specifying when validation occurs and what validation logic is executed for a given event.

const usernameValidation = ({ value, trigger }) => {
  // Event agnostic validation (sync)
  if (!value) {
    throw Error("Username is required");
  }

  // Validation on submit (async)
  if (trigger == "submit") {
    return isUsernameTaken(value).then((taken) => {
      if (taken) {
        throw Error("Username is already taken");
      }
    });
  }
};

In this above example, you can see we are able to run more expensive async validation exclusively on validation events such as submit.

Submission

Complimenting the new validation events, there's a new useSubmit hook which can be used for completion and progression in a form.

See the new submission guide in the docs

const { isValidating, hasSubmitted, handleSubmit } = useSubmit(() => {
  console.log("Submit validation succeeded!");
});

handleSubmit(); // Trigger submission

It returns a handleSubmit function which guards submission logic from being called until validation completes.

There's also additional state for tracking the status of async submit validation (isValidating) and for tracking whether the handleSubmit function has been called (hasSubmitted).

Combined with validation events, this hook provides all the necessary tools to do complex submit-specific validation without ever having to introduce your own state.

const usernameValidation = ({ value, trigger }) => {
  if (!value) {
    throw Error("Username is required");
  }

  if (trigger == "submit") {
    return isUsernameTaken(value).then(taken => {
      if (taken) {
        throw Error("Username is already taken");
      }
    })
  }
}

const Form = () => {
  const [usernameProps, usernameMeta] = useField({
    name: 'username',
    initialValue: '',
    validate: usernameValidation  
  });

  const { isValidating, hasSubmitted, handleSubmit } = useSubmit((values) => {
    fetch('/submit-form', {
      method: 'POST',
      body: JSON.stringify(values),
    });
  });

  return (
    <div>
      <input type="text" {...usernameProps} />
      <button onClick={handleSubmit}>
        {isValidating ? <Spinner /> : "Submit"}
      </button>
    </div>
  );
}

Get started

Whether you're new or looking to update, head on over to the docs site to get started and be sure to check out the new live examples.

Hopefully, you found this interesting! If you have any thoughts or comments, feel free to drop them below or hit me up on twitter - @andyrichardsonn

Disclaimer: All thoughts and opinions expressed in this article are my own.

We need to talk about state in React

Andy Richardson — Thu, 10 Sep 2020 13:51:36 +0000

Everyone loves an opportunity to slam on the big dog of frontend - React; but when it comes to state, it really is first class!

Let's talk about state and explore how to make managing it a breeze.

Literal forms of state

First off, it helps to understand the two forms that state can take in an application.

Explicit state

In the case of modern React, this is useState and useReducer. Explicit state doesn't just come out of thin air - it has to be explicitly created and managed.

Derived state

A psuedostate of sorts, derived state is a result of processing one or more states (explicit or derived).

const [input, setInput] = useState(); // Explicit state
const inputValid = useMemo(           // Derived state
  () => input && input.length > 6,
  [input]
);

Choosing types of state

Knowing whether to use explicit or derived state might seem challenging - but there's a really simple answer.

Always use derived state where possible

Forgetting to stick with the above rule can lead to redundant state.

Unlike redundant code, redundant state is a real problem that actually exists; and can have an impact on everything from performance to maintainability.

Spotting redundancy

If you've ever written something like the following - I know I have - you've probably been guilty of creating redundant state.

const [value, setValue] = useState("");
const [isValid, setIsValid] = useState(false);

useEffect(
  () => setIsValid(value && value.length > 4), 
  [value]
);

A useEffect call which immediately calls a setState function is almost always an example of state that should be derived.

It doesn't seem like the worst thing in the world, and on it's own, it probably isn't. That being said, if this pattern exists, there's a good chance it exists in many places and can lead to a larger problem.

useEffect hell

Most of us have been on a project that has gone through useEffect hell. Trying to fix that one bug but being unable to trace it because a single state change causes a flurry of new renders.

The thing with useEffect is it can cause a cascading number of state updates... which in turn, can cause subsequent useEffect calls. This isn't an issue with the function itself - it's an issue with excessive state.

Tips for managing state

If I had one piece of advice for managing state, it would be to keep it to a minimum... but I don't have one just one piece of advice - so here's some more!

Batch state updates

When multiple state updates are being called at one time, it's useful to batch these together into one call.

With batching

const [{ fetching, data }, setState] = useState({ 
  fetching: true, 
  data: undefined 
});

useEffect(() => {
  (async () => {
    const data = await getData();
    setState({ fetching: false, data })
  })()
}, []);

// State 1: { fetching: true, data: undefined }
// State 2: { fetching: false, data: 1234 }

Without batching

const [fetching, setFetching] = useState(true);
const [data, setData] = useState();

useEffect(() => {  
  (async () => {
    const data = await getData();
    setFetching(false);
    setData(data);
  })()
}, []);

// State 1: { fetching: true, data: undefined }
// State 2: { fetching: false, data: undefined } 
// State 3: { fetching: false, data: 1234 }

Batched updates don't just mean fewer renders, there will be fewer possible states to deal with; making testing and reproductions much simpler.

Remember, fewer renders means fewer state changes

Use fixtures

Fixtures (or stories) are an incredible tool for understanding, modelling, and documenting all the states of your app.

Find out more about fixtures over here.

Try useMemo more often

It's surprising how much of an impact it can make.

Hopefully, you found this interesting! If you have any thoughts or comments, feel free to drop them below or hit me up on twitter - @andyrichardsonn

Disclaimer: All thoughts and opinions expressed in this article are my own.

Getting a Browser Extension Through Review

Andy Richardson — Fri, 19 Jun 2020 10:47:41 +0000

Browser extensions can be confusing at the best of times, but recently we've seen that the current state of affairs is causing web store reviews to take weeks and there is evidence to show that nobody is safe from the wrath of the review teams.

Needless to say, whether you're publishing your first browser extension or already have one listed, it's worth making sure your submissions are prepared for a rigorous review.

Here's a quick run-through for how you can lock down your browser extension and increase your chance of getting your extension approved swiftly.

Timing your permissions

The fewer permissions your extension requires to run, the faster it's likely going to get through review. One of the ways you can trim down permissions without changing functionality is by only requesting additional permissions on an ad-hoc basis.

👎 Install-time permissions

The simplest approach, called "install-time permissions", involves an extension requesting, and being granted, permissions upon installation. Once installed, this allows the given extension to make use of its privileges at any time.

As you can imagine, this is a pattern that does little to restrict extensions functioning outside of their advertised feature-set. Most other ecosystems (see Android, IOS and W3) have tried to avoid or phase out this approach in favor of runtime permissions.

Consider an extension that granted install-time permissions for accessing all webpages and the microphone. Following installation, such an extension has the potential to do the following without requiring further approval:

Read any webpage content — bank details, messages, network requests
Manipulate any webpage content — inject adverts, political propaganda, Rick Astley videos
Listen to any user events — see key presses (including passwords), clicks, network traffic
Record audio at any time

Chrome install prompt for

Because permissions are granted in advance and the user has no way to regulate these permissions following installation, be prepared for meticulous reviews and publishing delays.

👍 Runtime permissions

Runtime permissions are a safer option for both users and extension developers. They work in a similar fashion to most other platforms by requesting no special permissions at install time and instead, prompt the user for additional permissions as-and-when needed.

Chrome install prompt for optional permissions

While this can take more time to implement, by using this method users can have a better idea as to when and why an extension is using any privileges. This also provides a means for the user to use a given extension with limited functionality, as opposed to the all-or-nothing case that install-time permissions offer.

The major caveat that comes with runtime permissions is the requirement that a permission request must be triggered inside of a user event handler. In other words, there is no way to programmatically trigger a permission request unless it is immediately following a user gesture.

Requesting webpage permissions

There are a few different ways a webpage can request permission to access a webpage — with some being better than others.

🥇 Active tab

For cases where a content script is required to run on the current active tab and only following a user invocation:

"permissions": ["activeTab"],

Example use of active tab permissions in an extension manifest.

Because this permission is technically always granted at runtime (by user invocation), declaring it as an install-time permission is common practice and will not result in any warnings to the user.

You might want to use this if:

your extension can be invoked by a user action
your extension only needs permission for the current tab

🥈 Match patterns

This is a host permission, intended for extensions where a content script is required to run on specific domains:

"permissions": [
  "https://my.domain.com",
    "https://*.domain.com",   // Subdomain wildcard
],
// or
"optional_permissions": [
  "*://my.domain.com",      // Scheme wildcard (https|http|ws)
  "https://my.domain.com/*" // Path wildcard"
],

Example use of scoped host permissions in an extension manifest.

You might want to use this if:

your extension needs permission for specific known URLs
your extension can't use activeTab

🥉 All URLs

This is the broadest host permission scope intended for extensions where a content script is required to run on all webpages without user invocation.

"permissions": ["<all_urls>"],
// or
"optional_permissions": ["<all_urls>"],

Example use of in an extension manifest.

You might want to use this if:

your extension needs permission for all URLs
your extension can't use activeTab

Preventing unsafe code

While reviewers will be spending a fair amount of time reviewing the code you submitted, that's not all that needs to be considered. There is likely a fair amount of code that is also included with your extension and has not been explicitly written by you (such as third-party dependencies).

Not only does your code need to be assessed for malicious content, dependencies are also an attack vector.

🛡 Content Security Policies (CSP)

Just like websites, browser extensions can make use of a Content Security Policy (CSP) to limit how code can be sourced and run in the browser.

"content_security_policy": "script-src 'self';",

Example use of a content security policy in an extension manifest.

While we haven't yet found an automated solution for injecting safe checksums into the extension manifest at build time, there is a webpack plugin to do this for webpages — an additional layer of security if your extension includes any HTML pages (such as a devtools panel).

🕵️‍♀️ Linting your extension

The team over at Mozilla have created a great CLI for browser extensions called web-ext. It includes a bunch of neat features for running, building, and testing your extension; and most importantly, it can lint extensions to flag vulnerabilities.

Output from running web-ext lint

From issues in the manifest to unsafe practices in your JS, this one catches a myriad of issues and is a must for anyone building a browser extension.

There's a chance that this will be used by store reviewers when assessing your extension so be sure to address any flagged issues before publishing.

Sealing the deal

For the time being, browser extension reviews are being done by humans, not an automated program. Reviewers are the number one people to please as they have the power to accept or deny your listing onto the web store, so be sure to make their life as easy as possible and get them on your side.

📘 Document the purpose

Try to document what the purpose of the extension is, and make it clear why the extension requests the permissions it does.

For example, a "dark mode" extension that accesses your webcam without explanation could trigger alarm bells. If it's explained that the webcam is used to detect ambient light, this is less likely to be the case.

🗺 How did you get there

Nobody likes reading minified code. If there is a build process to publish your extension, make sure to provide the source code and instructions on how to reproduce the built files.

Linking to the open source repo is another great way to gain trust.

🦺 Be conservative

A secure extension which uses CSPs, minimal permissions, and minimal dependencies should take less time to review.

This process isn't perfect

Even if you follow all these guidelines, there's no guarantee your project will be reviewed in a short amount of time. Manual reviews naturally take a long time and there are plenty of good reasons why some of the prior advice can't be followed.

Our very own urql Devtools is a great example of this. Like many other devtools extensions, anyone with our extension installed will have a content script run on every page they visit (i.e. install-time permissions for <all_urls> ).

Frankly, we don't want this to be the case, but the current state of browser extension permissions means it's just not possible to request permission to access a page at runtime when a user opens the devtools panel (as this does not constitute a "user gesture").

Can browser extensions be even safer in the future?

Making suggestions for browser standards is well outside of my skill-set and I'm sure there are some very smart people actively working on this.

One thing I would like to see is a continued movement towards reducing the number of browser extensions requesting permissions at install-time. The limitation in which extensions can only request permissions following a user gesture was clearly to prevent permission spam. I agree with the incentive but I suspect this decision has led to many extensions (including devtools extensions) being cornered into requesting more than permissions than they actually need.

Prevention of permission request spam in Android 10

Hopefully, in the future, we'll see a process that follows in the footsteps of other platforms that have managed to give users dynamic permission control while also preventing permission request spam.

Additional resources

Credits

Special thanks to Umar Hansa (@umaar) for the peer review!

This post was made thanks to the the support of Formidable. Check out the original post here.

urql Devtools: The Road to V1

Andy Richardson — Fri, 22 May 2020 16:48:00 +0000

In June 2019 we (the urql team) decided to set an adventurous goal of creating fully-featured developer tools for our GraphQL client.

10+ months, 10+ contributors, and 100+ pull requests later, we're proud to announce our first major release! Read below for the journey on how we got there or skip to the announcement.

📦 Setting up an extension

One of the biggest hurdles we encountered when creating our first devtools extension was the initial boilerplate. For good reason, web pages can't talk directly to extensions or devtools panels without a myriad of communication layers.

We looked into existing implementations but found that, given their maturity, there was a lot more going on than just messaging. Because of this we decided to take our own approach by making use of an EventTarget in our background.js to route messages from our browser windows to their respective devtools panel.

We've found that this simpler approach, while it does have caveats, has worked great for our use cases.

Having also received great feedback from others working on browser extensions in the community, keep your eyes peeled for a boilerplate project / tutorial from us in the near future!

💬 Sending messages from urql

Once messaging from the webpage to the devtools panel had been completed, it was time to put our money where our mouth is. urql has been created with extensibility in mind and we've been very vocal about this — so how did it do?

(Un)surprisingly well! Without modifications to the core urql client we were able to make an exchange which could:

See all incoming messages and responses
Inspect the state of the cache
Trigger GraphQL requests (sent from the extension)

This more than sufficed for our early pre-releases, but as time went on we found that we wanted more.

Introducing the debug target

Implementation details are an important part of any developer tool and over time it became apparent that we needed to find a way to expose the internal events happening inside of exchanges. A few examples include network triggers and responses (fetchExchange), cache invalidation (cacheExchange), and any other events which would be useful for debugging purposes.

In order to accommodate for this, we've added additional debugging capabilities to urql as of v1.11.x. It works a little something like this:

The urql client creates a debug source on creation
A dispatchDebug function is passed to every exchange
Exchanges can call this function to dispatch debugging events (at any time)
Anyone with access to the client can listen to these events
More details can be found in the new debugging section of the docs

For the devtools extension, this means that we can listen to debug messages coming from any exchanges and create a debugging experience for our users which is implementation agnostic - in other words, you could create a new exchange today and seamlessly have debugging info shown in the devtools extension just by calling dispatchDebug.

🛠 Building out the panel

At this point, we've now got messages coming from urql to the panel, so all we need to do now is create a webpage to present them... right? Well kind of...

Developer experience

In our first few months of working on the "frontend" of our extension, we found the developer experience to be rough, at best. Live reloading just wasn't an option, many changes would require us to reload the whole extension, and triggering/mocking debug events was a painful experience.

The biggest productivity booster, by far, was a few months in when we made the choice to develop using fixtures. This came with a whole host of benefits — from lowering the barrier to entry for new contributors, to faster updates thanks to the addition of hot reloading.

It also gave our team much more confidence in changes being made. We can quickly see the changes to fixtures while reviewing a PR and all our expected states are now modeled. To add to that latter point, fixtures allowed us to implement visual regression testing which would prove to flag unexpected visual and functional anomalies.

Design

Many of us working on this project have some kind of design experience in one form or another. As for design experience with browser extensions... not so much.

Data-heavy designs can be hard at the best of times; but add in the unique standards for browser panels such as smaller base font sizes, conservative use of spacing and color, and different interaction patterns - it doesn't take long to realize we're not in ~~Kansas~~ Bootstrap anymore.

While we're still learning, here are a few findings we've made on the way:

Acknowledge existing design languages - Chrome and Firefox have some great devtools so use them as a starting point
Aim for a highly static layout - theres going to be a lot of information on screen and having dynamic content such as panes appear and disappear can be more jarring than useful
Keep your information hierarchy flat - there's just not enough space for a deeply nested hierarchy (i.e. elements such as h1, h2, h3, etc) so flatter is better

🚀 The end result

After all that work, we're happy to announce that urql Devtools v1 is out! Here's what it does to make your GraphQL development experience even better!

Event timeline

Visualize all debugging events from your exchanges
Track queries, mutations, and subscriptions that have been executed
See network and cache updates, responses, and errors
Locate which components are triggering GraphQL requests

The events panel in urql Devtools v1.0.0

Cache Explorer

Explore an interactive overview of your cache
See which results are coming from the cache
Identify updates to cached data

The explorer panel in urql Devtools v1.0.0

Request tool

Trigger GraphQL requests directly via urql client
Easily change the state of your app
Explore your backend schema

The request panel in urql Devtools v1.0.0

🙏 We did it!

A special thanks to everyone in the community who helped make this happen!

Whether you contributed code, reported issues, created feature requests, or provided feedback — you've helped make the urql developer experience what it is! We look forward to seeing the ecosystem continue to blossom 🌻

To find out more or get involved, check out the resources below:

This post was made thanks to the the support of Formidable. Check out the original post here.

Going back to basics: Conditionals

Andy Richardson — Thu, 12 Mar 2020 12:43:21 +0000

If you're reading this, there's a good chance you learned how to use conditionals years ago.

If not, throw Error("learn conditionals")

Given how early on we learn about them, it's easy to overlook the impact of conditional management and how conditions can spiral out of control.

Below are a few real world examples of code I've seen in the wild and some tips on how to avoid these traps!

Avoiding reassignment

Starting off with the basics, if you're able to assign a value at declaration time, you have one less moving part in your code.

Here's a basic example where reassignment is used.

let x;

if (y > 5) {
  x = "large"
} else {
  x = "small"
}

In the following example, we remove the need for reassignment by using a ternary.

const x = (y > 5) ? "large" : "small";

If your language doesn't support ternaries, or you have more than two conditions to consider, you can opt for isolating your conditional logic into a function.

const x = getSize(y)

Note: A good exception to this rule is when iterating through collections

Flattening conditionals

Another tip is to keep your conditions flat - even if this means reusing a value inside of your conditional check.

This trims down the number of conditions you need to work with.

Here's an example of nested conditionals.

if (x) {
  if (x > 5) {
    return "large"; 
  } else {
    return "small";
  }
} else {
  return "unknown";
}

And here is an identical function body but with flattened conditionals.

if (x && x > 5) {
  return "large"
}

if (x && x < 5) {
  return "small"; 
}

if (!x) {
  return "unknown"
}

This example segues us nicely onto this next tip...

Handling undesirable states early on

Anyone who has used JavaScript callbacks likely recognises this pattern

doSomething((err, data) => {
  if (err) {
    return handleError(err)
  }

  handleData(data);
})

By handling undesirable states early on with a conditional guard, we add a layer of safety to all successive code and remove the need to re-check.

We can apply this pattern to our flattening conditionals example

if (!x) {
  return "unknown"
}

if (x > 5) {
  return "large";
}

return "small";

Remember, the logic for this is identical to the earlier nested conditional example. Which do you think is easier to read?

Conclusion

These few tips are a solid guideline for how to break out your code and can have a substantial impact on reducing the number of moving parts and interleaving conditions.

When writing code with lots of conditional logic, remember to ask yourself

What are the different resulting states/actions that can occur
What conditions are required to meet these states
How can I implement them in a way that is mutually exclusive

TL;DR: Keep things static, return often, and take a moment to think about your logic.

Hopefully, you found this interesting! If you have any thoughts or comments, feel free to drop them below or hit me up on twitter - @andyrichardsonn

Disclaimer: All opinions expressed in this article are my own.

Fixture-First Development

Andy Richardson — Wed, 04 Mar 2020 10:43:19 +0000

When you hear the word "Storybook," you probably think UI libraries. Tools like Storybook and Cosmos have been around for a few years now and do a pretty awesome job of presenting UI components in isolation. What most don't consider, however, is how these tools can go beyond just presenting UI components.

Let's talk about this!

Thinking in state

Consider the typical Button component in a UI library. When designing or implementing this component, one of the key considerations we ask ourselves is

"What states will this button have?"

Things might start off with a few simple states such as default and disabled.

Then comes the interactive states such as hovered and active...

Then primary and secondary...

Then primary disabled hovered and secondary disabled hovered

Before you know it, you have many states to consider and keep tracking.

This is when creating fixtures (or stories) starts to provide some real benefit. A fixture is a way of fixing the state of a component and modelling it in a browser environment. By doing this, we document our many states and also provide a means for quickly reproducing them during development and testing.

Compositional components

Moving higher up the component tree, it's easy to lose this state-first way of thinking about components. As scope increases, the core responsibilities of components don't change

Rendering output
Triggering side effects

While fixtures don't always help us demonstrate side effects, we can always use them as a means of modelling state.

Working in isolation

One of the first pages in the official React docs - Components and Props - states the following.

Components let you split the UI into independent, reusable pieces, and think about each piece in isolation.

Somewhere along the way, I think we forgot this is why we, as a community, chose to use React and not a page-scoped solution such as jQuery.

While focusing on integration is clearly important, there's huge value in being able to change and test components in isolation.

What this looks like

Here's an example of a Page component that has many states that are dependent on a network request and its response:

Believe it or not, everything you see above was made in total isolation and without spinning up the full site. Note how we can simulate states of our GraphQL client such as fetching and error without any magic — just fixtures and state.

Because React allows us to think of each piece in isolation, there's much less overhead required to do front-end work than you might think. Sure, we eventually need to bring everything together, but that is a small part of the whole development process.

Creating Fixtures

Depending on what tool you choose to use, the way in which you create fixtures will differ, but the process will almost always be the same.

1. Find the component you want to work on

Each project is different but you'll likely want to create fixtures for macro-components such as pages, forms, cards and modals.

For this example let's assume we're working with a page component that makes a GraphQL request and presents the state of that request to the user.

export const PostsPage = () => {
  const [getPostsState, refetch] = useQuery({
    query: gql`
      query GetPosts {
        posts {
          id
          title
          content
        }
      }
    `
  });

  if (getPostsState.fetching) {
    return (
      <ContentCentered>
        <Spinner />
      </ContentCentered>
    );
  }

  if (getPostsState.error) {
    return (
      <ContentCentered>
        <Icon type="warning" />
        <h1>Error</h1>
        <p>{getPosts.error.message}</p>
      </ContentCentered>
    );
  }

  if (getPostsState.data.posts.length === 0) {
    return (
      <ContentCentered>
        <Icon type="empty" />
        <h1>No posts found!</h1>
      </ContentCentered>
    );
  }

  return (
    <Content>
      {getPostsState.data.posts.map(
post => <PostCard key={post.id} {...post} />
            )}
    </Content>
  );
};

2. Setup the props and contexts for all key states

Once a component has been decided, it's time to work out what key states would be useful to have in a fixture. In our case, the key states of this page component are

Fetching
Error
Empty list
Populated list

Here's an example fixture that mocks the key states noted above for the PostsPage component:

const fetchingState = {
  executeQuery: () => { fetching: true },
};

const errorState = {
  executeQuery: () => { error: new Error("Something went wrong") },
};

const emptyState = {
  executeQuery: () => { data: { posts: [] } },
};

const dataState = {
  executeQuery: () => { data: { posts: [{ id: 1, name: "My post" }] } },
};

export default {
  fetching: (
    <GraphqlProvider value={fetchingState}>
      <PostsPage />
    </GraphqlProvider>
  ),
  error: (
    <GraphqlProvider value={errorState}>
      <PostsPage />
    </GraphqlProvider>
  ),
  empty: (
    <GraphqlProvider value={emptyState}>
      <PostsPage />
    </GraphqlProvider>
  ),
  data: (
    <GraphqlProvider value={dataState}>
      <PostsPage />
    </GraphqlProvider>
  )
}

Since hooks have replaced high-order components, you're going to find yourself mocking contexts more often, so get used to it!

Note: Most libraries don't document how to mock their context so you might have to dive into some code (or do some console.logs) to find out what the different states of the context look like.

We've now added dedicated documentation on mocking out context to the urql docs and encourage other library authors to do the same!

3. Develop within those fixtures

Once your fixtures are in place, you can test, style, and change logic within components quickly and without distraction! 🎉

Fixtures can also be used for automated testing such as visual regression, component snapshots, and functional testing.

Be mindful of changes that should be tested on a site-wide deployment, such as changes to network requests, hooking into new context, or just adding a component to the site for the first time. As mentioned earlier, this won't be too often, but in these cases, integration testing is the way to go.

Finding out more

Hopefully, if you've gotten this far, you're interested in trying this out for yourself!

I've put together an example repo that contains source code and live examples (including those used in this post) demonstrating the use of fixtures in a real-world project.

Examples include:

Fixtures of network requests & responses
Fixtures of modals, forms, and validation
Fixtures of ui components
Visual regression testing (using... you guessed it, fixtures)

Also, a huge shoutout to the contributors of the project React Cosmos who have made a great tool and documentation on developing with fixtures!

This post was made thanks to the the support of Formidable. Check out the original post here.

DEV Community: Andy Richardson

Packages don't solve distributed systems

The scaling problem 📈

Using packages 🎁

Coupling 👫

Runtime coupling

Code coupling

Ownership 🙋

Making it work 🤔

Service Oriented Architecture 🏗

Bounded contexts 🔲

Inevitable coupling 🚨

The internet

It can still go wrong

Ownership 🙋

Conclusion

GraphQL requests over HTTP/s are slow

Traditional transport mechanisms

Managed sockets

Improved performance (in theory)

Testing the theory

Performance results

Making the switch

Conclusion

GraphQL pagination with DynamoDB - Putting it together

🐙 Creating a resolver

Determining direction

Querying the Database

Page sizing

Constructing Edges

Correcting edge order

Constructing PageInfo

Cursors

Next pages

Final result

🚀 Conclusion

GraphQL pagination with DynamoDB - DynamoDB pagination

🏎️ Traversal

Inverting traversal order

📚 Paging

🔑 Cursors

Exhausting a query

Cursor construction

GraphQL pagination with DynamoDB - Cursor specification

💬 Pagination grammar

Edge

PageInfo

Connection

🎨 Creating a pageable field

🏎️ Page traversal

Forward pagination

Arguments

Retrieval

Backward pagination

Arguments

Retrieval

How I exploited NPM downloads... and why you shouldn't trust them

🔮 The illusion of downloads

What is a download

Registries

🧑‍💻 Exploiting the system

Creating a script

Triggering the Lambda

🚀 The result

Download stats don't work

"Popularity"

Conclusion

Whats new in Fielder V2

How we got here

New features

Validation events

Submission

Get started

We need to talk about state in React

Literal forms of state

Explicit state

Derived state

Choosing types of state

Spotting redundancy

useEffect hell