DEV Community: Frachtwerk GmbH

Mastering Software Versioning in JavaScript Projects

Ferdinand Mütsch — Mon, 09 Nov 2020 09:52:42 +0000

Introduction

A frequently overlooked aspect of software development is the proper versioning of code. Consistent and descriptive version numbers not only help developers keep track of their own work, but can also inform users of your software about what to expect from a new release. While versioning is especially important for libraries and frameworks which other projects depend on, benefits apply to standalone applications equally.

In this article, we introduce techniques, conventions, and tooling that helped us establish a robust way of versioning our JavaScript- and/or TypeScript-based software projects.

Concepts

Semantic Versioning

One of the most important aspects to think of when it comes to versioning is the version number itself. Before caring about tooling and others, you need to come up with syntax and semantics for it.

A concept that is well established among open source software projects is Semantic Versioning, or SemVer. When following this specification, a version number consists of three digits separated by dots, like 1.5.4 or, more formally <MAJOR>.<MINOR>.<PATCH>, where each individual part has a meaning:

MAJOR: Incrementing it indicates that there have been fundamental changes to the software and the new version is most likely not backward-compatible with the previous one.
MINOR: Essentially indicates that new features were added, but backward-compatibiltiy is still guaranteed.
PATCH or BUG FIX: Gives a hint that minor changes or bug fixes had been introduced recently.

Strictly following these semantics helps to maintain a common understanding of what a certain version means and what to expect from a new release.

Conventional Commits

The second concept that we committed ourselves to follow is Conventional Commits. Similar to semantic versioning, the conventional commits specification provides common syntax and semantics for information provided by a developer. However, in this case, the convention is not about the version number itself, but about the commit messages composed by developers when checking in new code into version control. The goal is to standardize their format and make them machine-readable to a certain extent.

When following conventional commits, a commit message essentially has to be prefixed with one of a few keywords.

fix: – A commit message with this prefix indicates a bug fix
feat: – A commit message with this prefix indicates the introduction of a new feature or functionality
refactor: – A commit with, whose message is prefixed like this, contains code refactorings, i.e. internal, technical modifications of the implementation of certain logic
chore: – This prefix indicates minor, miscellaneous changes of any type, that do not necessarily affect the user immediately
BREAKING CHANGE!: A commit message with this prefix warns about comprehensive, fundamental changes and indicates that the newly released version is likely to be incompatible with the previous one

The conventional commits specification comprises a few more keywords and also allows developers to come up with custom ones. However, these are the most relevant ones.

Tooling

When having paid attention, one might have recognized a few similarities in the semantics of conventional commits and semantic versioning. Commits with fix-changes correspond to the PATCH version, feat goes well with the MINOR version and BREAKING CHANGEes will inevitably result in a new MAJOR version.

As a consequence of following the above conventions, we enabled our project for an automated versioning workflow.

standard-version CLI

standard-version is a JavaScript tool that utilizes conventional commits to automatically enforce semantic versioning. Moreover, it is capable of automatically generating a changelog in Markdown format, which developers can provide their users with.

When running standard-version, the tool scans your commit history since when it was last executed, searches for fixes, feats, or breaking changes, and adapts the project's version accordingly.

To add it to an existing project, all you need to do is:

Install it as a dependency

$ yarn add -D standard-version  # (or npm i --save-dev standard-version)

Optionally add it as an NPM script to your package.json

{
    "name": "your-cool-project",
    "version:": "0.0.1",
    ...
    "scripts:" {
        "release": "standard-version"
        ...
    }
    ...
}

Release Workflow

After the development team has committed to consequently follow the conventional commits specification and all tooling is set up, a typical workflow to release new versions of your software might look like so.

Once a new version is ready to be released, i.e. at the end of a sprint, a developer executes yarn release (or npm run release) to kick off standard-version. As a result ...

... the project's commit history is scanned to determine which part of the version number needs to be incremented
... the version property of the project's top-level package.json is set to the new version
... a CHANGELOG.md file is written, containing separate sections for features and bug fixes
... the changes are committed to Git
... the new commit is given a Git tag corresponding to the new version

Depending on your setup, a push to the remote repository might kick off your CI/CD workflow, which may automatically build a new Docker image with the newly introduced tag and push it to a public or private registry. Using tools like Watchtower, the new image might even be rolled out to production automatically.

The only manual steps required in this workflow were a single yarn release command and a Git push. Nothing more, nothing less.

Conclusion

The above workflow has proven to be a convenient and consistent way of managing and releasing new versions of our JavaScript- and TypeScript-based frontend-, backend- and library projects and is even more beneficial with proper CI/CD pipelines and tooling like GitLab, Docker, Watchtower, Portainer, and others. It might even be adapted to projects written in other programming languages.

Modern, reactive web APIs with GraphQL, Go and Server-Sent Events – Part 2

Ferdinand Mütsch — Mon, 08 Jun 2020 00:00:00 +0000

In the previous part, concepts and benefits of GraphQL and GraphQL Subscriptions were presented and proposed as a modern, highly flexible alternative for designing web APIs.

In this post, we are going to look at actual code. For demonstration purposes, a small single-page application (SPA) is built as an example. It can serve as a cleanly structured starting point for new apps based on the proposed tech stack.

muety / go-graphql-sse-example

Basic example application using Go + GraphQL Subscriptions + Server-Sent Events + MongoDB + VueJS

go-graphql-sse-example

Basic example application to demonstrate an alternative way of building web APIs compared to REST. It combines these technologies:

Go as the primary backend-side language
GraphQL as a "protocol" for defining web interfaces
Server-Sent Events as a simple protocol for live updates, used as an implementation of GraphQL Subscriptions here
MongoDB as a flexible document database for storage
VueJS as a frontend framework to build single-page web applications

This project is intended to serve as a starting point to build modern, GraphQL-based single-page web application with a clean, opinionated code structure in backend and frontend.

Original blog post

Inspired by:

Requirements

Go >= 1.13
NodeJS >= 12.8.x
A MongoDB database
- You can get a free one at mlab.com

Usage

Backend (`/`)

Edit config.yml to configure your database
Set up the database (i.e. add users, collections and demo data)
Compile the GraphQL schema (located in schema…

View on GitHub

What to build?

The demo web app built in the context of this article comprised of a client-side frontend, built with VueJS and a server-side component built with Go. The two components interact with each other through a GraphQL API, which additionally offers the option to subscribe to data updates using Server-Sent Events.

With this demo app, we aim to mimic the functionality of a very basic, minimalist food ordering system. Customers can choose from a list of food products and place their orders. They are shown a waiting number an estimated processing time for their orders and are notified once the order is ready to be picked up. One the other side there is the kiosk operator, who views a live dashboard of currently pending orders, which appear just as they are being placed.

Technology Stack

In summary, the following technologies are used:

Go as the primary backend-side language
GraphQL as a "protocol" for defining web interfaces
Server-Sent Events as a simple protocol for live updates, used as an implementation of GraphQL Subscriptions here
MongoDB as a flexible document database for storage
VueJS as a frontend framework to build single-page web applications

In addition, graphql-request is used as a little helper library on the frontend to issue GraphQL queries more easily. It constitutes a light-weight wrapper around the plain Fetch API.

On the backend side, GraphQL development is facilitated by the excellent graphql-go package, which already provides well-defined guidelines to get started (thanks to the authors!).

Data Model

With GraphQL, you need to specify a schema using a GrapQL-specific syntax. It includes all entities, which your API should be able to deal with and is essentially a set of type definitions, split among one or more .graphql files.

Every GraphQL app consists of root types, defining all supported queries, mutations, and subscriptions as well as entity types.

For our app, the root schema looks like this (schema/schema.graphql):

schema {
    query: Query
    mutation: Mutation
    subscription: Subscription
}

type Query {
    product(id: ID!): Product
    products(): [Product]
    order(id: ID!): Order
    orders(status: String): [Order]
}

type Mutation {
    createOrder(order: OrderInput!): Order
    updateOrder(order: OrderUpdateInput!): Order
}

type Subscription {
    orderCreated(): Order
    orderChanged(id: ID!): Order
}

The type definition for the Product type referenced in the root schema is given as (schema/type/product.graphql):

type Product {
    id: ID!
    name: String
    description: String
    price: Float
}

The entire schema can be found in the GitHub repository.

After having defined your schema, in the case of graphql-go, it gets transformed into Go code, so it can be compiled into the final Go executable. It can then be loaded on application startup (see server.go) and served via an HTTP endpoint. Pretty straightforward!

graphqlSchema := graphql.MustParseSchema(schema.GetRootSchema(), &resolver.Resolver{})
http.Handle("/api/query", middleware.AddContext(ctx, &middleware.GraphQL{Schema: graphqlSchema}))

Resolvers

After having defined and loaded the schema, so-called resolvers need to be defined. The concept of resolvers is common among most GraphQL server libraries for various different programming languages. A resolver is responsible for providing the data for every field of an entity, e.g. for the ID, name, description, and price fields. Such fields can be literals, as it is the case with all fields of the Product type, but also other nested entities, like in the products field of the following Order type:

type Order {
    id: ID!
    ...
    products: [Product]
}

For literals, the resolver simply fills in the actual string, number, or boolean. For complex types, it delegates their resolution to their respective resolvers recursively. For instance, to resolve products, the orderResolver will ask a productResolver to do its respective duty.

When working with go-graphql, a simple resolver looks like this (resolvers/product.go):

type productResolver struct {
    p *model.Product
}

func (r *productResolver) Id() graphql.ID {
    return graphql.ID(r.p.Id)
}

func (r *productResolver) Name() *string {
    return &r.p.Name
}

func (r *productResolver) Description() *string {
    return &r.p.Description
}

func (r *productResolver) Price() *float64 {
    return &r.p.Price
}

Very simple. It is instantiated with a reference to a Product struct, which was previously loaded from the database and simply maps attributes of the raw data model to the respective GraphQL fields. Please note that it doesn't have to be that trivial. For instance, your MongoDB schema might define firstName and lastName fields, while your GraphQL type only has name. In that case, the resolver would have to do some basic concatenation.

Things get a little more complex when dealing with non-literal fields, like the Order's products above. In our database schema, an order only holds a list of product IDs as items. However, the GraphQL schema declares to return actual product objects. To do so, the respective resolver method first fetches the products for every ID from the database and then passes it on to productResolvers (see resolvers/order.go):

...
func (r *orderResolver) Products(ctx context.Context) (*[]*productResolver, error) {
    l := make([]*productResolver, len(r.o.Items))

    products, err := ctx.Value(service.KeyProductService).(*service.ProductService).GetBatchMap(r.o.Items)
    if err != nil {
        return nil, err
    }

    for i, id := range r.o.Items {
        l[i] = &productResolver{p: products[id]}
    }

    return &l, nil
}
...

Resolvers are very modular, coherent in themselves and can be composed together to build up an entire API. Note that even the very entrypoint of the GraphQL API is just a "root" resolver. You can find all resolvers, including those for mutations and subscriptions as well, in the repo.

Subscriptions

Probably the most interesting part here is subscriptions, as they provide a nice mechanism to make web applications reactive to backend-side data updates. As mentioned in the previous article, the GraphQL specification does not dictate how to technically implement subscriptions. Therefore, we decided to use Server-Sent Events (SSE) as a server-to-client communication channel. Technically, SSEs are simply a long-running HTTP request, which data is written to in form of a text stream and therefore very light-weight and easy to use.

On the backend side, we introduce a light-weight event bus to our services. For every data change, i.e. updates, creations or deletions, an event is published to the bus (see services/order.go):

func (s *OrderService) Create(order *model.Order) (*model.Order, error) {
    ...
    order.Id = res.InsertedID.(primitive.ObjectID).Hex()
    s.Hub.Publish(eventhub.Message{
        Name:   KeyOrderCreated, // order.create
        Fields: eventhub.Fields{"id": order.Id},
    })
    ...
}

Inside the resolver responsible for orderCreated queries, a subscription to the event bus is made once the user requests that subscription.

func (r *Resolver) OrderCreated(ctx context.Context) (chan *orderResolver, error) {
    c := make(chan *orderResolver)
    go subscribeOrder(service.KeyOrderCreated, ctx, c)
    return c, nil
}

func subscribeOrder(key string, ctx context.Context, c chan *orderResolver) {
    srv := ctx.Value(service.KeyOrderService).(*service.OrderService)
    sub := srv.Hub.NonBlockingSubscribe(10, key)

    defer func() {
        srv.Hub.Unsubscribe(sub)
        close(c)
    }()

    for {
        select {
        case <-ctx.Done():
            return
        case m := <-sub.Receiver:
            if u, err := srv.Get(m.Fields["id"].(string)); err != nil {
                log.Println(err)
            } else {
                c <- &orderResolver{u}
            }
        }
    }
}

When a new order is inserted via the respective service, the above method is triggered as a consequence of being subscribed to order.create events. It reads the orer's ID from the event, uses the qualified service to fetch it from the database and passes it on to an orderResolver to translate it into the schema-conformal format.

The HTTP handler, which dispatched the user's GraphQL query to the above resolver, in turn, has subscribed to the result Go channel and writes every incoming Order instance to always-open HTTP stream (see middleware/graphql.go):

...
c, err := h.Schema.Subscribe(ctx, params.Query, params.OperationName, params.Variables)
...
for r := range c {
        response := r.(*graphql.Response)
        responseJSON, err := json.Marshal(response)
        ...
        fmt.Fprintf(w, "data: %s\n\n", responseJSON)
        flusher.Flush()
        ...
}
...

Running Queries

Using GraphiQL

During development, queries against a GraphQL API can be issued using the interactive GraphiQL browser, as demonstrated here.

Programatically

On the client-side of our application, GraphQL queries are run to consume the API. After all, any client, that is able to speak HTTP, can also consume a GraphQL API, as GraphQL requests are really just POST requests with a certain query in the body.

For instance, the Vuex store action responsible for loading a list of products in the frontend is this (see store/products.js):

async fetchProducts({commit}) {
    const q = `{
        products() {
        id
        name
        description
        price
        }
    }`

    const data = await Vue.$api.graphql.request(q)
    commit('addProducts', data.products.map(Product.new))
}

For subscriptions, we rely on a slightly modified version of sse.js. It acts as a minimal wrapper around the browser's standard EventSource and allows us to run SSE requests as POST (instead of GET) queries and pass some data in the body. We do so to fit well with GraphQL, where queries are always POST requests with a respective query body.

Subscribing to new orders in the frontend is mostly done like this:

async subscribeOrderCreated({commit}) {
    const q = `subscription {
        orderCreated() {
            id
            queueId
            createdAt
            updatedAt
            status
            eta
            totalSum
            products {
                id
                name
            }
        }
    }`

    const source = Vue.$api.sse.request(q, null)
    source.addEventListener('message', e => {
        const payload = JSON.parse(e.data)
        ...
        commit('addOrder', new Order(payload.data.orderCreated))
    })
    ...
    source.stream()
}

Demo

And here's a live demo in action!

Outlook

While the current demo implementation serves as a – in our opinion – clean and well-structured starting point for building GraphQL-based web apps, its current state has one major drawback. It lacks authentication and authorization. Usually, you want to control which user can query and modify which data. Therefore, another blog post will follow in the future, which explains how to authorize GraphQL query endpoints.

Apart from that, we consider the present technology stack a promising choice for new web applications with the potential to facilitate clean, well-organized code and to make development easier, and more flexible.

Modern, reactive web APIs with GraphQL, Go and Server-Sent Events – Part 1

Ferdinand Mütsch — Mon, 08 Jun 2020 00:00:00 +0000

Introduction

In the course of this two-part article, the interested reader is briefly introduced to the basic of GraphQL and how it compares to traditional approaches. In the second part, an example single-page web application (SPA) is built to demonstrate the use of GraphQL in combination with further modern web technologies. The final project provides a clean, opinionated code- and project structure for both backend and frontend and constitutes a good starting point for new apps based on the presented tech stack.

muety / go-graphql-sse-example

Basic example application using Go + GraphQL Subscriptions + Server-Sent Events + MongoDB + VueJS

go-graphql-sse-example

Basic example application to demonstrate an alternative way of building web APIs compared to REST. It combines these technologies:

Go as the primary backend-side language
GraphQL as a "protocol" for defining web interfaces
Server-Sent Events as a simple protocol for live updates, used as an implementation of GraphQL Subscriptions here
MongoDB as a flexible document database for storage
VueJS as a frontend framework to build single-page web applications

This project is intended to serve as a starting point to build modern, GraphQL-based single-page web application with a clean, opinionated code structure in backend and frontend.

Original blog post

Inspired by:

Requirements

Go >= 1.13
NodeJS >= 12.8.x
A MongoDB database
- You can get a free one at mlab.com

Usage

Backend (`/`)

Edit config.yml to configure your database
Set up the database (i.e. add users, collections and demo data)
Compile the GraphQL schema (located in schema…

View on GitHub

What is GraphQL?

GraphQL is a relatively new (proposed in 2015 by Facebook engineers) approach to designing APIs for (web) backend applications and can be considered an alternative to REST or remote-procedure-call (RPC) mechanisms like JSON-RPC. In other words, it's "an open-source data query and manipulation language for APIs" [1]. The specification is open-source and actively evolving on GitHub. While REST APIs are currently the de-facto standard on the web (although not necessarily all of them being fully mature) – GraphQL starts to gain traction.

Comparison with REST and RPC

In contrast to REST, which is primarily structured around resources or entities and RPC-based APIs, which focus on actions or methods, GraphQL is all about the underlying data itself. The consumer of an API – usually the frontend / client-side part of a SPA – only has to know the schema and structure of the data provided by the API to CRUD it. Compared to REST APIs, where the consumer heavily depends on the fixed data structure delivered by the backend API, this is especially beneficial as it introduced a lot more flexibility and decoupling and can save the developer some time making the client-side application tolerant. Also, you will probably not need backends for frontends anymore.

Essentially, with GraphQL, the consumer asks exactly for what it needs and how it needs it, i.e. your client application tells the backend exactly what to return and in which format. Consuming a GraphQL API is like querying a database, but with more guidance and control.

Example

Let's look at an example to get a better idea of how GraphQL works, especially in comparison to the REST principles.

Imagine you have an e-commerce application with products and orders. Every order consists, among others, of a set of products. As the operator of the web shop, you might want to get a list of all orders. With a more or less RESTful API (we neglect the hypermedia controls in the example though), your request-response pair could look like this:

Request
-------
GET /api/orders

Response Body
-------------
[
    {
        "id": 125,
        "customerId": 8977,
        "createdAt": "2020-06-06T13:40:49.038Z",
        "productIds": [ 49863176 ]
    }
]

So far so good, but potentially you will also want to view the actual products right away. What you got are only ids, for each of which you would have to issue another API call to retrieve it. Alternatively, the API could also return nested objects, like so:

[
    {
        "id": 125,
        "customerId": 8977,
        "createdAt": "2020-06-06T13:40:49.038Z",
        "products": [
            {
                "id": 49863176,
                "name": "Slim T-Shirt navy-blue",
                "price": 17.90,
                "options": [
                    {
                        "id": "size",
                        "name": "Size",
                        "description": "T-Shirt size",
                        "values": [
                            {
                                "id": "s",
                                "name": "Size S",
                            },
                            {
                                "id": "m",
                                "name": "Size M",
                            },
                            {
                                "id": "l",
                                "name": "Size L",
                            }
                        ]
                    }
                ]
            },
        ]
    }
]

However, that is — to my understanding – not truly RESTful anymore. Also, while the above example is still quite straightforward, things get ugly as nested objects include other nested objects, that include other nested objects, that... Quickly you get JSON responses of several tens or hundreds of kilobytes, although you're potentially only interested in two or three attributes. Moreover, on some pages of your shop you may be interested in all possible options (e.g. "size") of a product, but not on others. Should your API define different view models now and expose different endpoints? Or a single endpoints with query flags like ?expanded=true? Soon you might be catching yourself tailoring your API specifically to the needs of your client while neglecting REST conventions and a straightforward design.

With GraphQL, things are different. Your API is a bit dumber and less opinionated now and does not deliver data in a fixed structure, according to a specified GQL query, which looks a lot like JSON. The above example might look like this, now:

Request
-------
POST /api/graphql/query

{
    "query": "\{
        orders {
            id
            customerId
            products {
                name
                price
                options {
                    name
                }
            }
        }
    \}"
}

Response Body
-------------
{
    "data": {
        "orders": [
            {
                "id": 125,
                "customerId": 8977,
                "products": [
                    {
                        "name": "Slim T-Shirt navy-blue",
                        "price": 17.90,
                        "options": [
                            {
                                "name": "Size",
                            }
                        ]
                    }
                ]
            }
        ]
    }
}

This way, you get only the data you want. All your API has to know is how to fetch every piece of data. All your client has to know is how the data schema itself looks like.

Try it out

GitHub's official API offers GraphQL query endpoints. You can try it out using their GraphQL explorer.

GraphQL Basic

Since this article does not aim to be another introduction to GraphQL, you can read most of the basics about fields, data types, etc. in the official docs. However, it is worth mentioning that GraphQL supports three types of queries:

Query: "Standard" type of queries, used for fetching data (see above). Similar to what you would do with a GET in REST.
Mutation: Query type used to modify data. Similar to what you would do with a PUT, POST, PATCH or DELETE in REST.
Subscription: Query type to communicate your intent to subscribe to live data updates.

Subscriptions

While a basic GraphQL application will at least use the former two types, the latter is especially interesting in the context of this article. Using subscriptions, you can have your web frontend be notified when new data arrives at the server or existing data changes. For instance, the operator of the above web shop could have a live-updating dashboard, that shows new orders just as they are placed.

For subscriptions, the GraphQL standard does not define a lot more than their plain existence and purpose. Especially, it is not defined how and which technology to implement them. On the web, any publish/subscribe-like mechanism that provides bi-directional or uni-directional server-to-client communication is appropriate. For the sake of simplicity, Server-Sent Events are used in this article.

What's next?

This part gave a brief introduction to GraphQL. The next part is about actual code. We're going to build an example web app with live-updates using GraphQL, Go, MongoDB and VueJS.

Modern, reactive web APIs with GraphQL, Go and Server-Sent Events – Part 2

Ferdinand Mütsch for Frachtwerk GmbH ・ Jun 8 '20

#graphql #go #mongodb #vue

DEV Community: Frachtwerk GmbH

Mastering Software Versioning in JavaScript Projects

Introduction

Concepts

Semantic Versioning

Conventional Commits

Tooling

standard-version CLI

Release Workflow

Conclusion

Modern, reactive web APIs with GraphQL, Go and Server-Sent Events – Part 2

muety / go-graphql-sse-example

Basic example application using Go + GraphQL Subscriptions + Server-Sent Events + MongoDB + VueJS

go-graphql-sse-example

Requirements

Usage

Backend (/)

What to build?

Technology Stack

Data Model

Resolvers

Subscriptions

Running Queries

Using GraphiQL

Programatically

Demo

Outlook

Modern, reactive web APIs with GraphQL, Go and Server-Sent Events – Part 1

Introduction

muety / go-graphql-sse-example

Basic example application using Go + GraphQL Subscriptions + Server-Sent Events + MongoDB + VueJS

go-graphql-sse-example

Requirements

Usage

Backend (/)

What is GraphQL?

Comparison with REST and RPC

Example

Try it out

GraphQL Basic

Subscriptions

What's next?

Modern, reactive web APIs with GraphQL, Go and Server-Sent Events – Part 2

Ferdinand Mütsch for Frachtwerk GmbH ・ Jun 8 '20

Backend (`/`)

Backend (`/`)