DEV Community: Murat Çorlu

Front-end may soon need new sub-disciplines

Murat Çorlu — Sat, 23 Jul 2022 19:14:53 +0000

Backend effort on most projects is declining every day with the help of high abstraction by cloud services. We can already make complete client-side applications with dynamic and collaborated data by just using some public services (like Firebase or CloudKit). We can manage authentication without having any server to maintain. Many authentication providers and even some authentication services allow us to do that.

And obviously, things will not stop here. It’s not hard to imagine that we’ll have more database services where we can directly write/read data from our front-ends securely and efficiently. The backend work of most of the projects will be just connecting and configuring multiple cloud services to each other.

So, will every web developer be a “front-end developer”? Of course, there will always be a need for more low-level server-side specialists, at least for creating those cloud services. But maybe 90% of the coding effort for web projects will be done on the client side. And this will mean the “front-end” will have more and more new responsibilities and challenges.

We can already see those with the new web APIs every day we start to hear, like Web Assembly, Web Worker, Web GPU, etc. The work we’ll begin to handle those new layers will be not just “basic UI” things. Instead, we’ll move some new stuff to the “front-end” that we previously handled on our backends with the help of the new power we’ll have in browsers.

Take Web Assembly, for example. Web Assembly allows us to run some libraries in the browser with native performance. You cannot (and intend to) do DOM manipulations inside the Web Assembly layer. Still, you can make very CPU-intensive things like image processing, data cryptography, or even maybe Machine Learning.

Web Workers gives us to run some other JavaScript routines outside of our render cycles without blocking UI. WebGPU will give us the full power of GPU to make even more efficient graph calculations, crypto calculations, or 3D renderings. WebRTC allows it to create a peer-to-peer connection with another user of our client app and send/receive any data, including streaming video or audio. File System Access API gives full access to the local file system of the user to read/write files and directories.

These are not very familiar things that we do daily as front-end developers. In the meantime, real UI work effort is not getting less. Creating durable, reusable, flexible, easy-to-use, and accessible components is still a big challenge. Providing the best UI with the best UX needs neat work with designers and product owners.

As a result, I expect we’ll need to consider splitting those new responsibilities inside web client-side development. Let me name them “Web UI” and “Web Core.”

Web UI Developers

Maybe we can roughly list the responsibilities of Web UI developers as follows:

Web Components
CSS
Routing
Forms
Animations
Accessibility
Web Fonts
Canvas
SVG, SVG filters

These are the items that we are very familiar with as front-end developers. They all are directly related to UI and user interaction. And we all know that if you consider being a specialist in all of those areas and being able to create a very well-implemented web application, there will be many challenges waiting for us.

This area is also where our popular web frameworks are responsible. We make components with Lit, manage complex forms with Angular, or create responsive grids with Tailwind. (The last one was a joke)

Web Core Developers

Web Core developers may be mainly responsible for the business logic and service-like backend part of the client app. These things will not directly change UI but primarily data management, business logic, and scalability (very familiar things for current backend developers, right?). If we want to list some technology names for Web Core developers, we may record:

WebGL - WebGPU
Worker
Web Assembly
Web Crypto
IndexedDB
File System Access API
WebRTC

There is no React or Angular here. Nothing here for them. But for complex web applications, those APIs are like treasures. Also, asking WebGPU or WebAssembly expertise from a senior React developer will not be fair.

Example project team structure

Let’s consider a project as a photo/video management web app. We’ll make a web application to add our photos and videos to a library, and edit, categorize or share them.

We’ll not have a backend code for this. We’ll use a database and a file storage service, which can work with an authentication service. Those services will be fully-managed, external services. We’ll handle image/video editing entirely on the client site. And we’ll store all data on our storage services in an e2e encrypted way.

As you can see, there are pretty many things to do for both UI and “Core” sides in this project. For UI:

A fancy UI to list, search and add many photos and videos to the library.
A fully-featured and easy-to-use UI for editing images and videos
Share albums with others, comment on photos
Making this app “accessible” as much as possible

On the other side, there are also huge challenges on the core side:

Writing some “Web Workers” to communicate with our cloud services in a non-blocking manner
Writing a very efficient image manipulation library
Writing a very efficient video editing library
An API to crypt/decrypt images and videos before sending them to our storage service and after fetching them
Storing some part of the data locally to make our app run offline and sync with services later

When we list our tasks like this, I think separating the types of tasks becomes very clear. “Web Core” tasks are pure IO challenges that never touch DOM or CSS. On the other hand, the APIs that the “Web Core” team will provide to the “Web UI” team will be very similar to what we currently have with the “Frontend - Backend” world. JS interfaces replace REST or GraphQL APIs.

Wrapping up

Titles and disciplines are not big walls to keep developers in their responsibilities forever. Instead, they are helpers focusing on one thing at a time. Distraction is one of the biggest diseases of our time. And I think it’s very valuable for every IT professional to focus on quality, not quantity.

Mocking REST endpoints in web apps, the easy way

Murat Çorlu — Wed, 01 Sep 2021 14:03:31 +0000

tldr; Use client-side mocks for your REST API endpoints with connect-api-mocker

Why we need mocks?

At one of my previous companies, in the early times, we had a flow like that:

A feature implementation is created by design team
Designs come to front-end team(us). We create static HTML pages, then task goes to back-end team.
Back-end team makes HTML dynamic.
Then probably because of we need some re-work and some extra JavaScript codes, the task returns back to front-end team.
Then task goes testing team…

After we changed our architecture as a Single Page Application and started to use REST API as the back-end, we wanted to make that flow faster and decided to use client side mocking solutions. The idea was to change our flow like that:

Design comes to front-end team
Front-end and back-end team decides endpoint structures that they will need for that task.
Front-end team uses mocks and starts implementation without waiting for real endpoint
Back-end team tries to create that endpoint. When endpoint is ready, front-end team doesn’t need any change for release. That means, you can develop your task by parallel with your back-end team. This will be improve your productivity and you will not have lot’s of troubles about your back-end dependencies.

This would be also good for reproducing interesting bugs. Sometimes we had some very exceptional but also annoying issues that are very hard to reproduce with real data. With an opportunity like defining API responses in a simple way, it would be very simple to create some awkward situations for testing.

Another advantage would have a development environment that doesn’t really need to connect some remote servers, even internet. You would not need a working REST API server to work.

Then we created connect-api-mocker

What is connect-api-mocker?

connect-api-mocker is an npm package (actually a simple connect middleware) that can be used for creating fake responses for REST endpoints.

We created the first version in 2013 at one of my previous companies and also shared on Github. After I left there, I forked the repository and continued to improve it to use it on my clients’ projects. I also created an npm package and added many features.

connect-api-mocker, uses some mock json files in filesystem that are created with same folder structure with REST endpoints. This makes it very simple to use and modify it. But it’s not just that, there are more!

How is it easy to use mocking

You can use connect-api-mocker with Webpack, Grunt, Gulp and other connect.js powered environments. You can install connect-api-mocker with command:

npm i --save-dev connect-api-mocker

To use it with a webpack project, you need to use setup property of webpack-dev-server:

var apiMocker = require(‘connect-api-mocker’);
...
setup: function(app) {
    app.use('/api', apiMocker('mocks/api'));
},
...

That configuration means, api mocker will try to respond requests that has url starts with /api by some files in the mocks/api folder. For example, if you make a GET request to /api/users/1 it will respond that request with file located in mocks/api/users/1/GET.json.

You already got the idea, right? The request mapping structure is like that:

[METHOD] /{baseUrl}/{path} > {basePath}/{path}/[METHOD].json

It will work all kind of request methods like POST, PUT, DELETE, OPTIONS etc. Some examples:

POST /api/users/1 ==> mocks/api/users/1/POST.json
DELETE /api/users/1 ==> mocks/api/users/1/DELETE.json

JSON files are normal, simple JSON files.

Custom responses

Creating mock json files with same path by request path is a good idea, but it’s not enough for all cases. If you also want to create some different mock responses for query or other request parameters, you will need more. This is where custom response support comes in.

Custom responses are simple connect middleware functions that also created in same path except(and of course) with a JS file extension. For example, instead of using simple POST.json file, you can create a POST.js file with content like that:

module.exports = function (request, response) {
    if (!request.get('X-Auth-Key')) {
        response.status(403).send({});
    } else {
        response.sendFile('POST.json', {root: __dirname});
    }
}

If you place that file to mocks/api/users/POST.js and if you make a POST request to /api/users, api-mocker will run your custom response middleware instead of returning POST.json file in the path. In the custom response example above, we are testing existence of a header and if it’s not there we are responding with a 403 error, if it’s there, we are simply sending POST.json file content in the same folder.
Can you see the power of that functions?

Another example:

GET /api/users?type=active ==> mocks/api/users/GET.js

module.exports = function (request, response) {
  var targetFileName = 'GET.json';
  // Check is a type parameter exist
  if (request.query.type) {
    // Generate a new targetfilename with that type parameter
    targetFileName = 'GET_' + request.query.type + '.json';
    // If file does not exist then respond with 404 header
    if (!fs.accessSync(targetFileName)) {
      return response.status(404);
    }
  }
  // Respond with targetFileName
  response.sendFile(targetFileName, {root: __dirname});
}

With an example like above, we can send separate JSON responses depending on a query parameter.

And another example with another good opportunity:

GET /api/gateway/3213132 ==> mocks/api/gateway/3213132/GET.js

var requestCount = 0;
module.exports = function (request, response, next) {
    requestCount++;
    if (requestCount == 1) {
        response.json({
            status: 'notready'
        });
    } else {
        response.json({
            status: 'ready'
        });
    }
};

Because of connect servers are running as an instance, we can store some in-memory variables like the example above. In that example we are responding different result depending of request count to that endpoint.

As you can see, you don’t have any limit with custom responses!

Wildcard support

We need one more thing: defining some wildcards in path definitions. We need that because it’s not very cool to create folders for every ids of our models like users/1, users/2, users/3 etc. What is about to create o folder like users/__user_id__ and use that user_id variable in our custom responses as a request parameter? Yeah, that would be great! The good news is it’s already there!

So if you make a request something like that:
GET /api/projects/50b3c102–298a-45ad-acad-e21b6c1bbdcc/

You can define a mock in mocks/api/projects/50b3c102-298a-45ad-acad-e21b6c1bbdcc/GET.json If api mocker finds a file with that path, it will respond with that. But another option is creating a file in a path like mocks/api/projects/__project_uuid__/GET.json. In that case, all requests to projects/{something} that has not a specific mock file, will be responded with same response. Also it’s possible to define a custom response in the same folder with a GET.js file and produce a dynamic response depend on wildcard variable that defined in folder name( project_uuid ). For example:

module.exports = function (request, response) {
  response.json({
    id: request.params.project_uuid
  });
}

Wildcard definitions are starting and ending with double underscore. And it’s the last that api mocker checked. So looking for a mock response flow is like that:

look for mocks/api/projects/50b3c102-298a-45ad-acad-e21b6c1bbdcc/GET.js
look for mocks/api/projects/50b3c102-298a-45ad-acad-e21b6c1bbdcc/GET.json
look for mocks/api/projects/__{something}__/GET.js
look for mocks/api/projects/__{something}__/GET.json

Also you can define multiple nested wildcards for a path like: mocks/api/__resource_type__/__resource_uuid__/GET.json. That will be respond all GET requests to api/*/*.

NextOnNotFound setting

By default, if you enable api mocker for a path and you don’t define a mock file for a request, it will be respond with a 404 Not Found error. But it’s also available to set nextOnNotFound to true then all not found requests will be passed to next middleware. Sample configuration:

app.use('/api', apiMocker({
  target: 'mocks/api',
  nextOnNotFound: true
});

This is very powerful if you only want to mock some part of your endpoints instead of all of them. With the help of proxy features of Webpack, Grunt or Gulp, you can continue to get responses from your real api that you don’t want to mock. An example for webpack:

// webpack.config.js
...
setup: function (app) {
  app.use('/backend/api/v1/', apiMocker({
    target:         'mocks/api',
    nextOnNotFound: true
  }));
  app.use('/analytics/api/v1/', apiMocker({
    target:         'mocks/analytics',
    nextOnNotFound: true
  }));
},
proxy: {
  '/backend': {
    target: 'https://api.backend.yourdomain.com'
  },
  '/analytics': {
    target: 'https://api.analytics.yourdomain.com'
  }
}

In the example above, we defined 2 api mocker for 2 separate paths and also defined some proxies to real api servers for that paths. With the help of nextOnNotFound option, requests to that paths that doesn’t have a mock definition will be passed to that proxies.

Conclusion

I used connect-api-mocker nearly on all my projects for last 4–5 years with several teams and we were very happy for getting advantages of it. I also suggest to use it or some solutions like that in your projects to simplify and make fast your development environment. Because of it’s also open source, you are very welcome for your contributions.

muratcorlu / connect-api-mocker

Connect middleware that creates mocks for REST APIs

Do you have an idea or a problem with api mocker? Just create an issue on Github repository and let’s talk about it.