DEV Community: tnodell

Creating A New Data Provider In Graphweaver

tnodell — Tue, 19 Dec 2023 23:40:50 +0000

Recently I was working on creating a new data provider for Graphweaver, our open source tool for connecting multiple data sources. Since there may be others in the future creating a data source from scratch I thought it would be valuable to document the process here.

Step 1: Create a Data Provider

Creating a new data provider requires extending the base graphweaver provider. This defines how data providers interact with Graphweaver. You will need to extend this class to implement your new provider. All of the base data provider methods can be found in these docs.

This example only implements the find and findOne functions, which gets entities and is used on the adminUI on the dashboard and details panel, respectively.

src/packages/core/src/base-provider.ts

export class Provider<D, G> implements BackendProvider<D, G> {
    constructor(readonly backendId: string) {
        if (!backendId) throw new Error('BackendId must be defined');
    }

    // READ METHODS
    public async find(
        filter: Filter<G>,
        pagination?: PaginationOptions,
        additionalOptionsForBackend?: any
    ): Promise<D[]> {
        throw new Error('Find not implemented.');
    }

The new provider accepts a generic DataEntity and GraphQLEntity (from graphweaver/core). It will include a private property of your base entity for the service you’re integrating with (more on the base entity and service in a bit). A constructor method with the backendId calls the provider class with the entity that is passed in. You can implement the remaining CRUD methods from the Graphweaver Base Provider. Those methods will call the service that interfaces with the new data source you’re integrating with.

export class YourBackendProvider<
  D extends BaseDataEntity,
  G extends GraphQLEntity<D>
> extends Provider<D, G> {
  private entity: YourEntity<D>;

  constructor(backendId: string, entity: YourEntity<D>) {
    super(backendId);
    this.entity = entity;
  }

  public async findOne(filter: Filter<G>): Promise<D | null> {
    const yourService = new YourService(this.entity);
    return yourService.getEntity(String(filter.id));
  }

public async find(filter: Filter<G>): Promise<D[]> {
    const salesforce = new Salesforce(this.entity);
    return salesforce.getEntities(filter);
  }

Step 2: Create a Base Data Entity

Now we’ll need to to define the base data entity representing the new data source. This will implement the BaseDataEntity type in Graphweaver. The BaseDataEntity includes methods to determine if the entity is a reference or a collection.

Your entity will have three properties:

new (data: any): D; Any class using this interface must have a constructor that takes some data and returns an instance of your data entity.
entityName: string; You must provide a name for the entity
getProperties: () => string; You must provide a method that gets the properties of the entities and returns them. In this case we return a string that goes into constructing a GraphQL query to our new data source.

Your base entity class is an abstract class that implements the above type.

export interface YourEntity<D extends YourBase> {
  new (data: any): D;
  entityName: string;
  getProperties: () => string;
}

export abstract class YourBase implements BaseDataEntity {
  static entityName: string;

  static getProperties() {
    // return what you need to get back data from your new source
   return `
        Id
        Name { value  }
    `;
  }

  public isCollection(fieldName: string, dataField: any) {
    return false;
  }

  public isReference(fieldName: string, dataField: any) {
    return false;
  }
}

Next we’ll create an example entity from this base. This entity will be mapped when we return data from our service.

import { Base } from "./base";
interface ExampleEntityData {
    id: string;
    name: string;
}

export class ExampleEntity extends Base {
    static entityName = "Example"; //This goes into the GQL request
    id!: string;
    name!: string;

    constructor(data: ExampleEntityData) {
        super();
        this.id = data.id;
        this.name = data.name;
    }
}

Step 3: Create a Service

Now we can implement our connection to our new data source. This service will reference your base entity. It’s up to you to implement the functions that will request data from your data source. In this example let’s make a query to some GraphQL endpoint. When the data returns, we’ll map it to the entity we defined earlier.

export class YourService<D extends Base> {
  private entity: YourEntity<D>;

  constructor(entity: YourEntity<D>) {
    this.entity = entity;
  }

  public async getEntities(filter: YourFilter<D>) {
    // Get the entities
    const query = `query {
                    result: ${this.entity.entityName}${filter ? constructFilter(filter) : ""} {
                        edges {
                            node {
                                ${this.entity.getProperties()}
                            }
                        }
                    }
        }`;

    const response = await fetch("your-endpoint.com", {
      method: "POST",
      headers: {
        "Content-Type": "application/json",
        Accept: "application/json",
        Authorization: `Bearer ${process.env.TOKEN}`,
      },
      body: JSON.stringify({
        query,
      }),
    });

    const data = await response.json();

    // Map new Entities from the data
    return data.map((data) => {
      return new this.entity({ id: data.id, name: data.name });
    });
  }

Conclusion

Now you have created a new data integration with Graphweaver. By following these three steps – creating a data provider, defining a base data entity, and implementing the connection to the service – developers can extend Graphweaver's capabilities to integrate with any data source.

Exploring The Benefits of GraphQL Code Gen

tnodell — Wed, 23 Aug 2023 00:34:56 +0000

GraphQL is the leading method of querying APIs in modern web development. Yet, developers face a common challenge using GraphQL: managing the types and structures of the data returned by their queries. Dealing with undefined values and casting partial types to make the Typescript compiler happy is frustrating. Or constructing types for each kind of response. But, there's a solution that alleviates this pain, boosts development speed, and enhances code quality – GraphQL Codegen. This powerful open-source tool automates the generation of code for GraphQL APIs. GraphQL Codegen produces types, queries, and hooks from the GraphQL schema. No more manual repetitive coding or uncertainty about types. In this post, we'll explore how GraphQL Codegen works, and how we use it in our open-source tool Graphweaver to facilitate a silky smooth developer experience. So let's dive in and uncover how this tool is reshaping the way we build applications.

When I started using GraphQL as a React developer, one of the things that frustrated me was that I didn’t know the type of data that is being returned from my query. My Typescript types were all empty unless I was casting the type of the response. Which meant I had to maintain a separate types definition file on the frontend. But this did nothing to solve the problem of what was being returned in the query. In the past, a partial solution to this was to use partial types everywhere client side. As a result, we’d have to check for undefined throughout our codebase, losing the benefits of a typing system. For example:

const GET_TODOS =  `
  query getTodos {
    todos {
      id
      description
      assignee {
        id
        firstName
        lastName
      }
    }
  }
`

const { data } = useQuery(GET_TODOS);

const todos = data?.todos as Partial<Todo>[]

const firstAssigneeFirstName = todos[0].assignee?.firstname

This creates a couple of gross code smells. First, we have to cast the Todo type because we don’t know what a todo is from the query. The type cast as Partial<Todo>[] is also a Partial which means we also have to handle anything in Todo being missing such as the assignee?. Alternatively, we could create a new type that explicitly Omits or Picks the exact type returned by the server. But this creates overhead in the form of keeping these types in sync with what’s being requested. Any changes to the query or component cascade into maintaining the correct type.

The solution is to automatically generate your types from the GraphQL schema, AKA GraphQL Codegen.

GraphQL Codegen helps automate the process of writing code for GraphQL APIs. By looking at a GraphQL schema we can generate code for our types, queries, and React hooks. This means that developers can save time by not having to manually write repetitive code. GraphQL Codegen also helps with type safety and consistency. It ensures that the code is strongly typed, and can be easily updated when changes are made to the API. This leads to faster development cycles and fewer bugs in production.

Because GraphQL document fields are already typed, it’s pretty annoying to try to keep your types in sync with them on the client. That’s why GraphQL code generation is so valuable. The gold standard for this process is GraphQL Code Generator.

How GraphQL CodeGen Works

GraphQL Code Generator is an open-source tool that generates code from your GraphQL schema. It’s highly configurable for your environment with a schema-first approach. The GraphQL schema serves as the contract between the frontend and backend of a GraphQL API. It defines the data types, relationships, and operations that clients can perform on the data. Using this, GraphQL Code Generator can determine the types, queries, and mutations on the frontend.

Generating the Typescript types from the GraphQL schema occurs in a two-step process. First, GraphQL Codegen looks at the types as defined by the schema. Then it creates a JSON data structure via the introspection query provided by GraphQL. This describes the relationship between your types and queries.

Using this JSON, GraphQL Codegen creates the Typescript types with a templating engine. This iterates through the custom JSON and creates the types and hooks that your frontend uses.

In practice, this looks like:

Installing the package
Setting the configuration
Executing the Codegen script

GraphQL Code Generator can not only create frontend typings but can create the backend resolvers from the GraphQL schema as well. This magic happens in the configuration.

const config: CodegenConfig = {
   schema: 'https://localhost:4000/graphql',
   documents: ['src/**/*.tsx'],
   generates: {
      './src/gql/': {
        preset: 'client',
      }
   }
}

schema identifies the URL for the GraphQL schema endpoint. documents define where your queries, mutations, arguments etc. are. The generates key describes the location where your types will output. And the preset is your template for what kind of output you expect. In GraphQL Code Generator, plugins handle the parsed GraphQL schema and describe the expected output. For example, the Typescript plugin generates the base Typescript types. Presets are groups of plugins for common configurations, such as the client preset, which includes the Typescript plugin. The client preset is configured for optimal usage with React or Vue and for GraphQL clients like urql and Apollo. Presets are opinionated and designed to get you up and running quickly.

GraphQL Code Generator is a powerful and fast tool for solving the problem of managing types from a GraphQL API. This is not solely limited to frontend web development but can generate your resolvers from the GraphQL schema as well! This approach is schema first, where developers can configure what they want their Codegen to output via a YAML file. The presets and plugins are defined in the YAML file.

Unfortunately, there are some limitations to having a schema-first approach to creating a GraphQL API. The generated code can be complex and hard to customise, and this philosophy has some inflexibility when custom logic is required that is not supported out of the box by the schema.

How We Use GraphQL CodeGen In Graphweaver

We use GraphQL Code Generator in our project Graphweaver. Graphweaver is a open source tool that allows you to connect multiple data sources, from databases, to rest APIs, to Saas Platforms, and expose those as a single GraphQL API. As a part of this process, we generate the GraphQL schema from the data source. From the schema, we then generate the types, queries and hooks for the frontend. Let’s go on a whirlwind tour of how we do this in Graphweaver.

It all starts with our source of truth in the data source. For this example let's imagine a Postgres database as our datasource. Following up with our Todos example above:

-- Create the Assignees table
CREATE TABLE assignees (
  id SERIAL PRIMARY KEY,
  first_name VARCHAR(50),
  last_name VARCHAR(50)
);

-- Create the Todos table
CREATE TABLE todos (
  id SERIAL PRIMARY KEY,
  description TEXT,
  assignee_id INT,
  FOREIGN KEY (assignee_id) REFERENCES assignees(id)
);

-- Insert sample data into Assignees
INSERT INTO assignees (first_name, last_name)
VALUES
  ('John', 'Doe'),
  ('Jane', 'Smith');

-- Insert sample data into Todos
INSERT INTO todos (description, assignee_id)
VALUES
  ('Complete project A', 1),
  ('Review documentation', 2);

Once you have a Postgres (or MySQL or SQLite) database with this up and running it’s easy to connect to Graphweaver and generate the entire backend and GraphQL for a project.

First, initialise the project:

npx graphweaver@latest init

This pulls down the latest version of Graphweaver from NPM and starts a new project. The cli will ask you for a name and what kind of database is being used. After you cd into the project and install the dependencies you can run another command to generate your backend.

npm graphweaver@latest import postgresql

This import command line tool takes the database and does its code generation to create the files for our backend entities and files. Currently, it supports Postgres, MySQL, and SQLite databases. In this case, running the import command opens a database connection to Postgres, fetches the database schema and builds metadata about the database. Then it generates the entity files, schema files, resolver files, and index files. Finally, it closes the connection and logs out what has been created. It’s quite handy to not have to do this manually! A full tutorial on connecting a database can be found here.

Whether we created this manually or with the import tool, we create the data entities, schema entities, and resolvers. Then you can pnpm start your project which kicks off the remaining code generation. When you initialise a new Graphweaver instance all the resolvers are passed into Graphweaver. The resolvers contain metadata about the entities and we use this to populate the TypeGraphQL metadata storage. Using this we generate the types, queries, and mutations using GraphQL Code Generation.

The result of this process entity types:

export type Todo = {
    __typename?: 'Todo';
    id: Scalars['ID']['output'];
    description?: Maybe<Scalars['String']['output']>;
    assignee?: Maybe<Assignee>;
    type?: Maybe<TodoType>;
};

Hooks which you can use like any other Apollo hook:

export function useGetTodosQuery(
    baseOptions?: Apollo.QueryHookOptions<GetTodosQuery, GetTodosQueryVariables>
) {
    const options = { ...defaultOptions, ...baseOptions };
    return Apollo.useQuery<GetTodosQuery, GetTodosQueryVariables>(TodosDocument, options);
}

And the return types from queries:

export type GetTodosQueryHookResult = ReturnType<typeof useGetTodosQuery>;

So getting back to our original example code. We can rewrite to:

const GET_TODOS =  `
  query getTodos {
    todos {
      id
      description
      assignee {
        id
        firstName
        lastName
      }
    }
  }
`

const { data } = useGetTodoQuery();

const todos = data?.todos

const firstAssigneeFirstName = todos[0].assignee.firstname

Where our data.todos are now typed from our React Hook useGetTodoQuery(). We no longer have to cast the type! Although we do have to handle the data itself being undefined during a loading state or error.

What’s more, is that if we change our query then our types update when we restart the server or have the Graphweaver watch command running. Say we don’t need to see the lastName of the assignee any more, the Graphweaver cli removes that from the response type. A file is generated next to each component ./component.generated which contains the types that the query is using. A complete example of what’s generated can be seen in one of our example repos on GitHub.

With a few commands, Graphweaver generates the backend components, schema, resolvers, and more. As the project progresses, Graphweaver automatically updates types based on query changes, ensuring type consistency. Ultimately, Graphweaver empowers developers to craft APIs that seamlessly amalgamate diverse data sources into a single, cohesive GraphQL interface. This results in a more efficient and structured development experience.

Summary

GraphQL Codegen is a tool that automates the process of writing code for GraphQL APIs by generating code for types, queries, and hooks from a GraphQL schema. It saves time, improves type safety and consistency, and can be used for both frontend and backend development. While there are limitations to a schema-first approach, we’re using it in Graphweaver to provide a code first solution for developers looking to get started quickly. But we also provide the flexibility to extend their API in any way they can code. We’re big fans of using GraphQL Codegen in our projects.

Get started using Graphweaver to generate your code from your database with our documentation to get you up and running in under 5 minutes. If you enjoy Graphweaver please give us a star on GitHub.