Document data stores are valuable for storing data in complex ways that may not fit into the rigid structure of a traditional database. They come with dynamic and self-describing schemas that offer you the flexibility to have documents with different fields in a collection. This ability to accommodate varying fields across documents eliminates the need for pre-defining schemas in a database. However, retrieving data from these databases can be difficult, as they’re not typically built for querying and searching data. Enter GraphQL: a flexible and efficient query language that can parse data from virtually any source.
Developing GraphQL APIs in the traditional way requires you to have a database entity defined, have a corresponding data access object created, depending on the queries it supports you need to have a layer that writes queries specific to the database for every entity and search parameter supported. A lot of developer time is spent on writing these layers that can become complex and hard to maintain overtime.
We at Intuit have developed a new translation library, which can convert any entity within a document data store into a searchable GraphQL API in just a few configurations. With our library, you can query and parse any data stored within these document data stores and provide it all within a GraphQL API, enabling you to create flexible, customised queries for your users.
Our library works by interrogating the schema of the document data store, extracting entity types and possible relationships. You can then configure these entities and properties with our easy-to-use configuration process, to create a GraphQL API that is tailored to your specifications.
The resulting GraphQL API is fully searchable, even for data that wasn’t searchable before. This enables you to provide customised search functionalities to your users, such as filtering, paging, and sorting. With our library, the schemas will always be up-to-date, as the translation process is automatic and immediate.
Developers can add this library to their existing stack with just a few simple steps. It’s useful for teams working with document-oriented data stores such as MongoDB, Couchbase, Elastic Search, DynamoDB, AWSDocument DB. Library is extensible for any new Document datastore and allows developers to specify the datasource to connect to, and can connect to multiple datastores for the same request depending on the entity requested.
The Power of Configuration: How Our Translation Library Translates Document Data into GraphQL APIs
Library is developed on top of the Spring Boot Framework. It reads graphQL schema from the configuration store and exposes the entities defined in Schema as a GraphQL endpoint.
It intercepts GraphQL Requests, parses the requests to identify the Requested Entity, Search parameters, Operation, Conjunction , Search values and fields requested by the end customer.
For the Search parameters passed in the request, It fetches corresponding fields on the document collection from the Configurations. These configurations are kept in-memory and any change in config is reflected instantly.
For instance,
customer-customerId=document.customer.identifier.id
is interpreted as -> if there is a search on Customer entity with CustomerId attribute, look for document customer and match customerId with path customer.identifiers.id
Depending on the operation, search parameters, conjunction, search values and configurations fetched, the library dynamically forms queries specific to a datastore configured in runtime.
Library allows users to specify the required datasource implementation.
customer-datasource=ElasticSearch
Library interprets that all queries on customer entity should be served from an ElasticSearch datasource and forms all Queries as required by Elastic search.
Library allows stitching the response from related collections. Depending on the requested fields, the Library reads related entities ,corresponding configurations and forms queries for every related entity as mentioned above and concurrently fetches data from all related entities.
customer-relations=products, subscriptions
This implies that Customer entity is related to Products and Subscriptions entities.
customer-relations-products=document.product.customerAccount.id
This implies that customer and product entities can be joined by matching customer.id and document.product.customerAccount.id fields.
Our translation library is compatible with any document database and achieves its generalisability by working with JSON document structures instead of relying on custom entity or data models.
Conclusion
Our translation library addresses the problems that developers face in building APIs for document data stores. It removes the need to write complex and database-specific queries for each entity and search parameter supported, freeing up developer time.
With this library, our teams are able to develop new APIs on Document data stores with No/Low code config driven approach.
APIs are now relatively easy to maintain, adding a new field to an existing entity, or adding a new entity, enabling search on any new field all of these need no code changes hence no Deployment and regression.
p.s. Will update the blog with GitHub project link once we successfully open source this Library.
Top comments (0)