How to handle pagination with Fauna

aargrk — Wed, 27 Jan 2021 15:25:31 +0000

Introduction

Writing resolvers in FQL (Fauna’s Query Language) is not the easiest thing on earth, especially if you are not familiar with this type of programming. At first sight it may look strange. The main thing I hear is "there are only functions and to do anything you have to have function inside a function inside a function..."

This article walks through how to get started with Fauna from a basic setup with default queries (built automatically when importing a schema) to creating a query with resolver recreating the default paginated resolver. Finally we will move to more complex resolvers that are accepting multiple filters.

Let’s start from creating a schema. Our data models are rather simple to not distract you from the resolvers and the implementation.

Chapter 1: Schema

Schema for the data we'll be working with.

type Student {
    name: String
    tests: [Test] @relation
}

type Project {
    name: String
    tests: [Test] @relation
}

type Test {
    name: String
    student: Student!
    project: Project!
}

We have 3 models. Student, Project and Test.

Each Test needs to be assigned to a Project. Student can take a Test, which later be linked to the Student.

If we upload the schema as it is right now, we will receive some queries out-of-the-box. Let's upload our schema.

Go to GraphQL tab and press import schema.

After uploading the Schema, some of the Queries and Mutations will be automatically created for us. We have 3 queries available - findProjectByID, findStudentByID and findTestByID.

All three receive data from a single document Project, Student or Test.

To get all Tests or Projects we need to add proper queries to the Schema:

# ...
type Query {
  projects: [Project]
  tests: [Test]
}

After uploading the schema we can also get all Tests and Projects

After above steps our schema looks like this:

type Student {
    name: String
    tests: [Test] @relation
}

type Project {
    name: String
    tests: [Test] @relation
}

type Test {
    name: String
    student: Student
    project: Project
}

type Query {
  projects: [Project]
  tests: [Test]
}

If you want to try each step on your own, it would be helpful to have some dummy data to work with. You can use this gist to add some fake data matching the above schema.

create a secret (in security tab) and replace the <fauna_secret>
don't forget to install packages fakerator and faunadb
run node generate.js

Chapter 2: Paginated queries

Right now we have 3 default queries allowing us to get data from a single document and our 2 queries to get Tests or Projects.

What if I'd like to get more specific data like Tests only from Project X?

I can either fetch the specific project and use the test field to obtain associated tests.

query TestByProject {
  findTestByID(id: <test_id>) {
    project: {
      data: {...}
    }
  }
}

While it looks "easy" It's not always the case, that's why we'll create a new query with a custom resolver.

Add new query to your schema.graphql file

type Query {
    ...
    getTestsByProject(id: ID): [Project] @resolver(name: "tests_by_project", paginated: true)
}

By using @resolver directive we specify that we want to use our resolver for this query. We pass the name of the Function that will be handling the request. paginated: true flag makes sure our new query behaves the same as the default ones. It paginates the data instead sending it all at once.

After updating the schema, new Function tests_by_project appears in the "Functions" tab. When you try to use the new query right now, you’ll receive an error: “Function X not implemented yet…”. So, let’s do it.

We’ll need to check if we have any Index that can handle such a query. We want to get all Tests matching the given Project ID. If you go to the Indexes tab, you'll see there is an already created Index with the name project_tests_by_project. It does exactly what we need.

In case you don't have this index or want to know how to create such, here is a script you can use in the console in Shell tab.

CreateIndex({
    name: "project_tests_by_project",
    source: Collection("Test"),
    terms: [
        {
          field: ["data", "project"]
        }
      ]
})

Now, we need to add some code to the tests_by_project Function. We would need to do basically two things, look for the Tests with given ProjectID and handle the pagination. Let's start with the first part.

Query(
  Lambda(
    ["projectID"],
    Let({
        project: Ref(Collection("Project"), Var("projectID")),
        match: Match(Index("project_tests_by_project"), Var("project")),
        data: Paginate(Var("match"))
      },
        Map(Var("data"), Lambda("ref", Get(Var("ref"))))
      )
   )
)

First argument the Lambda takes is the ProjectID our query looks for. Next, using Let() function, we define some of the variables that will clarify what the Lambda does step-by-step.

Under projectID we have stored a string representing the ID of the project. To filter by actual document, we’d need a Ref to the document, hence creating one under “project” variable.

What is under match variable looks for all documents satisfying the query and finally the "data" variable stores the documents. We need to use the Paginate function to "extract" the documents from the Set returned by Match(). In the next step, iterate over each document found and get its data.

The pagination. After adding the paginated flag to the resolver Lambda receives 3 additional arguments:.

size - specifies the number of documents returned in the single query
after / before - indicates where the query should start (both are returned with each query, so we can use "after' from last query, to get next set of data)

We can now pass them to the Paginate() function. The idea is to use each of those argument if it store any value, or skip if it does not:

Query(
  Lambda(
    ["projectID", "size", "after", "before"],
    Let(
      {
        ...
        data: If(
          And(IsNull(Var("after")), IsNull(Var("before"))),
          Paginate(Var("match"), { size: Var("size") }),
          If(
            IsNull(Var("before")),
            Paginate(Var("match"), { after: Var("after"), size: Var("size") }),
            Paginate(Var("match"), { before: Var("before"), size: Var("size") })
          )
        )
      },
      ...
    )
  )
)

Chapter 3: Displaying the data

To display the data we'll be using the react-table library. We would like to use the pagination query to get only the number of documents to be displayed on one page. To perform the API calls to fauna graphql endpoint, I'll use a react-query library with graphql-request.

Let's start with the basic configuration of those two and create "All Projects" page.

// AllProjects.js
import React, { useContext } from "react";
import { useQuery } from "react-query";
import { gql } from "graphql-request";
import Table from "./Table";
import { GraphqlClientContext } from "./App";

export default function AllProjects() {
  const { data, isLoading } = useProjects();

  if (isLoading) {
    return <span>Loading...</span>;
  }

  return <Table columns={columns} data={data} />;
}

function useProjects() {
  const graphqlClient = useContext(GraphqlClientContext);
  return useQuery("projects", async () => {
    const {
      projects: { data },
    } = await graphqlClient.request(
      gql`
        query {
          projects {
            data {
              _id
              name
            }
          }
        }
      `
    );
    return projects;
  });
}

const columns = [
  {
    Header: "ID",
    accessor: "_id",
  },
  {
    Header: "Name",
    accessor: "name",
  },
];

// Table.js
import { useTable } from "react-table";
import "./Table.scss";

export default function Table({ columns, data }) {
  const {
    getTableProps,
    getTableBodyProps,
    headerGroups,
    rows,
    prepareRow,
  } = useTable({
    columns,
    data,
  });
  return (
    <table {...getTableProps()}>
      <thead>
        {headerGroups.map((headerGroup) => (
          <tr {...headerGroup.getHeaderGroupProps()}>
            {headerGroup.headers.map((column) => (
              <th {...column.getHeaderProps()}>{column.render("Header")}</th>
            ))}
          </tr>
        ))}
      </thead>
      <tbody {...getTableBodyProps()}>
        {rows.map((row, i) => {
          prepareRow(row);
          return (
            <tr {...row.getRowProps()}>
              {row.cells.map((cell) => {
                return <td {...cell.getCellProps()}>{cell.render("Cell")}</td>;
              })}
            </tr>
          );
        })}
      </tbody>
    </table>
  );
}

// App.js
import React from "react";
import { BrowserRouter as Router, Switch, Route } from "react-router-dom";
import { QueryClient, QueryClientProvider } from "react-query";
import { GraphQLClient } from "graphql-request";
import AllProjects from "./AllProjects";

const queryClient = new QueryClient();
const graphQLClient = new GraphQLClient(`https://graphql.fauna.com/graphql`, {
  headers: {
    authorization: "Bearer <fauna_secret>",
  },
});

export const GraphqlClientContext = React.createContext();

function Main() {
  return (
    <Router>
        <Switch>
          <Route path="/projects">
            <AllProjects />
          </Route>
        </Switch>
    </Router>
  );
}

function App() {
  return (
    <GraphqlClientContext.Provider value={graphQLClient}>
      <QueryClientProvider client={queryClient}>
        <Main />
      </QueryClientProvider>
    </GraphqlClientContext.Provider>
  );
}

export default App;

That’s the basic setup we are going to begin with. You can find full repository here.

Current setup doesn't handle pagination at all, it displays only the first page of data. It's ok for some cases. (for example If I'll be sure I'll have only a few Projects available)

But in our case, I'll have a lot of Tests so I'd definitely want to use the benefits of server side pagination.

I'd like to be able to go back and forth with the data
I'd like to be able to change number of documents displayed per page

Let's start with extending the Table component with pagination controls.

We would be handling pagination by sending paginated request, hence we use useTable with the manualPagination option.

// Table.js
import React from "react";
import { useTable, usePagination } from "react-table";
import "./Table.scss";

const pageSizeVariants = [50, 75, 100];

export default function Table({
  columns,
  data,
  fetchData,
  loading,
  initialPageSize,
  pageCount: controlledPageCount,
}) {
  const {
    getTableProps,
    getTableBodyProps,
    headerGroups,
    prepareRow,
    page,
    canPreviousPage,
    canNextPage,
    nextPage,
    previousPage,
    setPageSize,
    // Get the state from the instance
    state: { pageIndex, pageSize },
  } = useTable(
    {
      columns,
      data,
      initialState: { pageIndex: 0, pageSize: initialPageSize },
      // We will be handling pagination by sending paginated request, 
      // not default client side, hence the manualPagination option
      manualPagination: true,
      pageCount: controlledPageCount,
    },
    usePagination
  );

  function changeSize(e) {
    setPageSize(Number(e.target.value));
  }

  React.useEffect(() => {
    fetchData({ pageIndex, pageSize });
  }, [fetchData, pageIndex, pageSize]);

  return (
    <>
      <table {...getTableProps()}>
        <thead>{headerGroups.map(renderHeaderGroup)}</thead>
        <tbody {...getTableBodyProps()}>
          {page.map(renderPage(prepareRow))}
        </tbody>
      </table>
      <div>
        <button onClick={previousPage} disabled={!canPreviousPage}>
          {"<"}
        </button>{" "}
        <button onClick={nextPage} disabled={!canNextPage}>
          {">"}
        </button>{" "}
        <select value={pageSize} onChange={changeSize}>
          {pageSizeVariants.map(renderOption)}
        </select>
      </div>
    </>
  );
}

function renderHeaderGroup(headerGroup) {
  return (
    <tr {...headerGroup.getHeaderGroupProps()}>
      {headerGroup.headers.map((column) => (
        <th {...column.getHeaderProps()}>{column.render("Header")}</th>
      ))}
    </tr>
  );
}

function renderPage(prepareRow) {
  return function (row, i) {
    prepareRow(row);
    return (
      <tr {...row.getRowProps()}>
        {row.cells.map((cell) => {
          return <td {...cell.getCellProps()}>{cell.render("Cell")}</td>;
        })}
      </tr>
    );
  };
}

function renderOption(val) {
  return (
    <option key={val} value={val}>
      Show {val}
    </option>
  );
}

We will require few additional props to pass to Table component:

fetchData - function that calls API to get data on every page/size change
initialPageSize - sets number of documents to display on first render
pageCount - initially, it indicates how many pages of data are available, we will not be able to get that information but we have to use it to control whether there is more data to display or not. react-table blocks pagination if current number of pages are the same as the pages count. We will increase the pageCount by one if there is more data, or keep the same if not.

Our component should properly react to the page and size change, and make a new request if any of those changed.

Let’s start with the query. We will be using the getTestsByProject. We need to define some query variables.

query($id: ID, $size: Int, $cursor: String) {
    getTestsByProject(id: $id, _size: $size, _cursor: $cursor) {
      data {
        id: _id
        name
        student {
          id: _id
        }
      }
      after
      before
    }
  }
}

size param to set number of documents to return in one query;
cursor param to indicate whether we want next set of data or previous one
after and before we will be passing one of them as a cursor param, to get next (after) or previous (before) documents.

As you can see there is no page param, so we cannot "tell" - give me documents from page 3. We can only operate in next/before manner. It adds additional complexity to our fetch*() method, but we’ll handle that..

That's the theory, let's write some code.

First I'll create a new hook - useTests()

// useTests
function useTests(projectID) {
    // react-table will send us the page index if user go back or next
  const [page, setPage] = React.useState({ index: 0, cursor: null, size: 25 });

  // we'll be using the GraphlClient to send requests
  const graphqlClient = useContext(GraphqlClientContext);
  const query = useQuery(
    [key, page.size, page.cursor, projectID],
    fetchProjects(graphqlClient)({ size: page.size, cursor: page.cursor, id: projectID })
  );

  return query
}

const fetchProjects = (client) => (variables) => async () => {
  const { tests } = await client.request(
    gql`
      query($id: ID, $size: Int, $cursor: String) {
        tests: getTestsByProject(id: $id, _size: $size, _cursor: $cursor) {
          data {
            id: _id
            name
            student {
              name
            }
          }
          after
          before
        }
      }
    `,
    variables
  );
  return tests;
};

useQuery hook will fire each time the page state changes.

And after adding some of the methods that will be used handle the pagination:

// useTests.js
function useTests(projectID) {
  ...

  // under query.data we have all the results from `tests` query
  // query.data -> { data, after, before }
    const tests = query.data?.data || [];
  const nextPageCursor = query.data?.after;
  const prevPageCursor = query.data?.before;
  const canNextPage = !!nextPageCursor;

  function nextPage() {
    if (!nextPageCursor) return;
    setPage((page) => ({
      ...page,
      index: page.index + 1,
      cursor: nextPageCursor,
    }));
  }

  const prevPageCursor = data?.before;
  function prevPage() {
    if (!prevPageCursor) return;
    setPage((page) => ({
      ...page,
      index: page.index - 1,
      cursor: prevPageCursor,
    }));
  }

  function changeSize(size) {
    if (size === page.size) return;
    setPage((page) => ({ index: page.index, cursor: null, size }));
  }

  function updateData({ pageIndex, pageSize }) {
    if (pageSize !== page.size) changeSize(pageSize);
    else if (pageIndex === page.index) return;
    else if (pageIndex > page.index) nextPage();
    else prevPage();
  }

  const canNextPage = !!nextPageCursor;

  return {
    ...query,
    data: tests,
    size: page.size,
    updateData,

    // page + 1 gives actual number of pages (page is an index started from 0)
    // Number(canNextPage) increase the pageCount by 1 if canNextPage == true
    pageCount: page.index + 1 + Number(canNextPage),
  };
}

If the user decides to go next - we want to fire the nextPage() method, if back prevPage() if only change size then changeSize() method. This logic lives inside the updateData() which will be fired after any page/size change.

Use new methods in Project component:

// Project.js
...
import { useParams } from "react-router-dom";

export default function Project() {
  const { id } = useParams();
  const { data, isLoading, pageCount, size, updateData } = useTests(id);

  if (isLoading) {
    return <span>Loading...</span>;
  }

  return (
    <Table
      columns={columns}
      data={data}
      fetchData={updateData}
      pageCount={pageCount}
      initialPageSize={size}
    />
  );
}

const columns = [
  {
    Header: "ID",
    accessor: "_id",
  },
  {
    Header: "Name",
    accessor: "name",
  },
  {
    Header: "Student",
    accessor: "student.name",
  },
];

// App.js
...
<Router>
  <Switch>
    <Route path="/projects/:id">
      <Project />
    </Route>
    <Route path="/projects">
      <AllProjects />
    </Route>
  </Switch>
</Router>
...

That allows the user to enter a page for each project. When a browser hits /project/<projectID> page Project component will be able to get the id from URL, using the useParams() hook.

Last change is to change the ID column on AllProjects table to render a link to a specific project page.

// AllProjects.js
import { Link } from "react-router-dom";
...

const columns = [
  {
    Header: "ID",
    accessor: ({ _id }) => <Link to={`/projects/${_id}`}>{_id}</Link>,
  },
  {
    Header: "Name",
    accessor: "name",
  },
];

And now looks like that's all - we have fully functioning paginated Table using paginated query 🚀

If you want to check the final solution, here is a link to the repository

but...
If you would like to take it a step further, instead of writing separate queries for each filter_by you want to use, there is a way to accept multiple filters in one query.

There is a high chance you would like to use filters in your query instead of multiple one-purpose queries, for example:

query {
    tests(filter: { 
    student: ["286712490662822407", "286712490702668289"], 
    project: ["286712490727835143"]
     }) {
      data {
        id: _id
        name
        student {
          id: _id
        }
      }
      after
      before
    }
  }
}

For that you will need to create (if not already exist) Indexes for each filter (tests by student and tests by project) and use them both when Paginate() the data. Example resolver with schema:

# schema.graphql
#...
input TestFilters {
  project: [ID]
  student: [ID]
}

type Query {
  # ...
  tests(filter: TestFilters): [Test] @resolver(name: "get_tests", paginated: true)
  #...
}

// get_tests.fql
Query(
  Lambda(
    ["filters", "size", "after", "before"],
    Let(
      {
        baseMatch: Match(Index("tests")),
        // creates match for every id in in filter.project array
        matchByProjects: Map(
          Select("project", Var("filters"), []),
          Lambda(
            "id",
            Match(
              Index("project_tests_by_project"),
              Ref(Collection("Project"), Var("id"))
            )
          )
        ),
        // creates match for every id in in filter.student array
        matchByStudents: Map(
          Select("student", Var("filters"), []),
          Lambda(
            "id",
            Match(
              Index("student_tests_by_student"),
              Ref(Collection("Student"), Var("id"))
            )
          )
        ),
        // combines all matches into one array
        // end up with [baseMatch, Union([projects]), Union([students])]
      match: Reduce(
          Lambda(
            ["acc", "curr"],
            If(
              IsArray(Var("curr")),
              If(
                // skips if empty
                IsEmpty(Var("curr")),
                Var("acc"),
                Append(Union(Var("curr")), Var("acc"))
              ),
              If(
                IsNull(Var("curr")),
                Var("acc"),
                Append([Var("curr")], Var("acc")),
              )
            )
          ),
          [],
          [
            Var("baseMatch"),
            Var("matchByProjects"), 
            Var("matchByStudents")
          ]
        ),
        intersectionMatch: Intersection(Var("match")),
        item: If(
          Equals(Var("before"), null),
          If(
            Equals(Var("after"), null),
            Paginate(Var("intersectionMatch"), { size: Var("size") }),
            Paginate(Var("intersectionMatch"), {
              after: Var("after"),
              size: Var("size")
            })
          ),
          Paginate(Var("intersectionMatch"), {
            before: Var("before"),
            size: Var("size")
          })
        )
      },
      Map(Var("item"), Lambda("ref", Get(Var("ref"))))
    )
  )
)

With that you are able to cover many requests with the same query and you have less functions to maintain.

I personally start with single-purpose resolver and switch to the multi-filter resolver when have many filter resolvers for the same Collection.

DEV Community: aargrk

How to handle pagination with Fauna

Introduction

Chapter 1: Schema

Chapter 2: Paginated queries

Chapter 3: Displaying the data