Joyce, a highly scalable event-driven Cloud Native Data Hub.

mogui — Thu, 15 Jul 2021 09:08:02 +0000

Years of blood and struggle, condensed in a few github repositories, finally public with a product emerged from different experiences inside the company I work!

This is Joyce

a highly scalable event-driven Cloud Native Data Hub.

Ok! Wait, what?

Joyce allows you to ingest data from (almost) any source and expose the ingested data as standard APIs (REST, event notification) automatically. In order to specify to Joyce which data we want to pick from the incoming data stream and how APIs will look like you need to describe the expected behaviour with a DSL based on json-schema.

Continue to read about it here : https://sourcesense.github.io/joyce-docs/

No REST for the wicked

mogui — Sun, 16 Jul 2017 18:50:20 +0000

This article (also read as rant) was heavy inspired by the reading of this article, which puts in words something that was floating in my mind a lot these years of making services talking between each others, being API, mobile client or whatever.

Why the fuck are we using REST~ish API everywhere?

if you don’t get this, we cannot talk aboutÂ music…

The Problem

It seems that nowadays a decent developer MUST know and use REST, but how we are doing it?

It’s pretty much clear to me implementing it or just consume it that there’s a lot wrong.

Main points are covered by the article cited at the beginning, but I’ll just remark some of them:

1. “RESTful APIs are usually tied to HTTP”

and makes use of it scattering logic around too much: headers, verbs, uris, response codes and body. This makes it difficult to debug and totally unintuitive, leading to random implementations.

When you want to use a 3rd party RESTxxx API you have to read the documentation because you can’t give anything for granted:

Do I need to provide an header or a token in the query string to authenticate calls?
How the header is named?
To create a resource do I use a POST or a PUT?
and for an update?
and for an upsert?
When an error occurs do I check only the status code or also the payload? (how many apis have the “status” field in the payload? I did it myself too, many times)
And the error message?
To filter out or query a list of resource which fields do I use in the query string?
And if I want something that isn’t a plain resource but a mixed collection?

2. “The REST vocabulary isn’t rich enough”

This is harshly true because REST is designed after the concept of resource, CRUD operations and linking them. It is a good model to represent linked data and managing it just like you can do on a RDBMS, but API are usually not only that.

Let’s just think of a mobile social app API where within the things you have to:

register a user -> PUT, POST user resource?
authenticate a user -> is it a resource???
create a new blog entry -> this I know!! POST /blog yeh!! Done. REST is awesome!
like a post -> mmm could be POST /blog/666, but I haven’t a payload! Can I use just a GET? and the action does really create a “like” resource or is it just a flag on the blog entry resource? should I use a PUT to update the blog entry resource? … wtf
search fora collection of resources -> GET /resource yes! search criteria and filters? queryString parameters? body payload? if that i need a POST but I am not creating nothing :( fuck wasn’t SOAP better? hell no! but wtf!
ask for a profile of a user but with more or few information -> query string parameter? another uri endpoint?
make new friend request -> ooh fuck!

It’s a mess and everyone is doing it in a different way, also REST related frameworks roll their own philosophy.

REST was a bless because it gave us freedom from the strictness of SOAP for example, but at what price?

Confusion I guess.

Whenever we have to design a REST~ish API, we have to answer too much questions and the problem is that there is no right answer.

Existing solutions

I am obviously not the only one having these feelings so there are people around trying to nail these points, author of the cited article propose his solution JSON-pure API. I think that it’s not so beautiful and usable, it kind feel going back to a SOAP~ish thing, but it has great ideas:

be agnostic of the transport, no more significant content attached to protocol specific things, like response code and headers, enable the use of whatever transport we should need: HTTP, Websocket, XMPP …
JSON is easy to debug

More stable and adopted things are out there: jsonapi, HAL, Mason, Collection json.

I think that each one either is not complete or just addresses the same resource/CRUD model, just in a different way.

What also I think is: we need a sort of standard to do standard things.

I think that there should be a best practice to do authentication for API, error handling and response format, things that from an API consumer point of view let you integrate more quickly.

SOAP, bad as it was, let you create clients by themselves that maybe it is too much, but it was intuitive to use, REST~ish things are not.

No, I don’t want to go back to SOAP :P so…

An idea…

The new idea is an old idea.

I don’t mean to write a new protocol specification.

xkcd

I think that the right things are already there but have just to be put together.

So what I want to write is more a “Specification of best practice” or a “Protocol that is just already-known-protocols composition”.

What it should be made of?

Since API are not just CRUD operation but more something like actions or procedure maybe a simple RPC is what we need. We already have a nice specification: JSON-RPC. It’s clean, simple as json, agnostic from the transport and there’s already a lot of clients libraries.
Requests authentication, where needed, should be embedded in this protocol and JWT I thinks is the right fit for that. Also RPC methods regarding authentication should be in the protocol specification standardising the jwt generation.
We can make a hazardous jump and have JSON Schema over the JSON-RPC payload to validate the API format, and with little effort have the API clients library autogenerated from the schema (hello again SOAP :3 ).

Are you sure you need RPC there’s this thing called GraphQL …

It’s been a while since I jot down these thoughts about REST, but never wrapped in an organic discussion, it seems I waited too long. It’s not yesterday that I have stumbled upon GraphQL, knowing it just superficially, I have seen it just as a wonderful way to retrieve data. A closer look made me totally reconsider RPC. With Mutations you can achieve anything, it is really just like an RPC declaring ahead what you want as a result, beautiful. What you miss from RPC is a uniformed error handling in the response, GraphQL doesn’t enforce anything in this regard. What doesn’t change from RPC is that GraphQL doesn’t deal with authentication too, so you have to take care of that by yourself, who said JWT??.

We can embed both in query/mutation, something like:

mutation {
  login(username: String!, password: String!) {
    token
    error
  }
}

query {
    myUltraAwesomeApi(token: String, aParameter: String) {
        error,
        result
    }
}

Conclusions

It’s time to put REST at rest, and make API by combine these old/new technologies that are really more appropriate in a many cases. So go out and play with RPC , GraphQL and JWT!

DEV Community: mogui