DEV Community: Rukshan J. Senanayaka

DX_1: How we created an Optimized Search for our Faculty's Career Fair Website

Rukshan J. Senanayaka — Mon, 19 Feb 2024 07:50:14 +0000

Last Update - 19th Feb, 2024 | 1347

DX_ Series

This is an article of my DX_ series where I post my developer experience one article at a time.

Introduction

The annual career fair organized by our Faculty of IT, University of Moratuwa stands as a vital bridge connecting students with top industry professionals. 2024 FIT Future Careers Career fair was held on 8th of February,2024.

Recognizing the need to enhance this connection, the student body INTECS responsible for the event has launched a dedicated website. However, the site's search feature initially fell short of expectations.

In this article, I'll share how I tackled these challenges to refine the search function, ensuring the website effectively matches the best talents with the greatest opportunities.

What was the problem?

Upon joining the career fair website team, my main focus was on deployment. However, I quickly identified a major flaw in the website's search functionality.

Originally designed to support 300 students, the site inefficiently fetched all student records in one API call for every load of the search page, performing the filtering on the client side.

This not only strained the website's performance but also risked overloading the API server, given our limited server resources. For instance, if even if the user views only the first page of the results, all 300 student details are already loaded, which is a huge overhead.

We faced a critical need for a more efficient search solution that balanced functionality with our budget constraints.

My thought pattern

Facing the search functionality issues on our career fair website, I recalled using Elasticsearch in a past project for efficient search and filtering. This experience inspired me to apply a similar solution, but cost was a concern due to our tight budget.

I explored open-source options, finding Typesense, a cost-effective, typo-tolerant search engine offering fast, relevant search capabilities and a managed cloud service with a free tier.

Despite initially considering self-hosting, their managed cloud service - Typesense Cloud proved to be the most practical choice, allowing us to enhance search efficiency without financial burden or extensive setup.

A little bit of problematic implementation

Earlier implementation looked something like this.

You can see when the Student.jsx component of the student search page is loaded, it loads all the users from the database via the API, which is not good.

useEffect(() => {
  axios.get(`user/get-all-users`).then((res) => {
    setAllUsers(res.data.data);
  });
}, []);

The data returned from the API endpoint are then used to update the state variable allUsers.

const [allUsers, setAllUsers] = useState([]);

The users were filtered using a simple JS filter() method. This frontend filter doesn't do much other than just simply filtering the already-retrieved database data.

This doesn't provide the performance improvements a dedicated search system would provide, such as typo-tolerance etc. multiple search criteria etc.

const filterUsers = (users, query, selectedAreas) => {
  return users.filter(user => {
    const fullName = `${user.f_name || ''} ${user.l_name || ''}`.trim().toLowerCase();
    const isNameMatch = fullName.includes(query.toLowerCase());
    let preferredAreas = [];
    try {
      preferredAreas = JSON.parse(user.preferred_area || '[]');
    } catch (e) {
      console.error('Error parsing preferred_area:', e);
    }
    const isAreaMatch = selectedAreas.length === 0 || preferredAreas.some(area => selectedAreas.includes(area));
    return isNameMatch && isAreaMatch;
  });
};

The solution

To improve the search functionality without changing the UI, I chose Typesense for its high performance and React compatibility. Typesense offers fast, typo-tolerant searches, ideal for instant feedback on user queries.

I integrated it using React InstantSearch components & hooks, adapting them for Typesense to enhance search without altering the UI. The setup involved initializing a Typesense server for the search index, facilitated by Typesense’s clear documentation.

This allowed for a smooth transition to Typesense’s optimized search components, ensuring easy customization and adaptability to our UI needs. Also automatic pagination and filter components are provided with this library.

A Node.js program was developed to create the indexes of the fields using which we needed to filter the students contained the schema definitions as below.

// Define schema for the 'students' collection
const schema = {
    name: "students",
    num_documents: 0,
    fields: [
        { name: "uni_index", type: "string", facet: false, optional: true },
        { name: "f_name", type: "string", facet: false, optional: true },
        { name: "l_name", type: "string", facet: false, optional: true },
        { name: "profile", type: "string", facet: false, optional: true },
        { name: "preferred_area", type: "string[]", facet: true, optional: true },
        { name: "skills", type: "string[]", facet: true, optional: true },
        { name: "cv_link", type: "string", facet: false, optional: true },
        { name: "front_picture_link", type: "string", facet: false, optional: true },
    ],
};

The following is how the indexes look in the managed cloud server.

Then in the react frontend application, we consumed the search and configured which fields and the weights we need to assign to each field during the searching.

export const typesenseAdapter = new TypesenseInstantsearchAdapter({
  server: TYPESENSE_SERVER_CONFIG,
  additionalSearchParameters: {
    queryBy: "f_name,l_name,skills,preferred_area,uni_index",
    query_by_weights: "3,3,2,2,1",
    numTypos: 3,
    typoTokensThreshold: 1,
  },
});

I tackled the challenge of integrating Typesense’s search components without altering the existing design by customizing React InstantSearch components to match the application's aesthetic.

By applying custom styles and utilizing modified hooks like a debounced useSearchBox and an adapted useHits, I ensured the search functionality blended seamlessly and efficiently with the UI.

This resulted in an enhanced user search experience—faster and more accurate—without disrupting the design continuity. The successful integration underscored my ability to improve functionality while maintaining the visual integrity of the application.

Who keeps the search index updated? - Cron!

The above mentioned Node.js application was deployed as a Cronb job to automate updating the Typesense search index, ensuring our database records were consistently synchronized for reliable search functionality.

This application, designed to run as a cron job, leverages Node.js's strengths in asynchronous operations and database compatibility, making it an efficient solution for data indexing.

It periodically (every 15 minutes) checks for new or modified database entries, formats them for Typesense, and uploads them to the search server on managed Typesense instance.

Deployed as a serverless function on Digital Ocean using do-functions and do-functions-server packages, this setup streamlines the indexing process, maintaining the search feature's efficiency and reliability. Otherwise even if the students update their details and save to database, those updated details would not be returned in search results without being indexed.

Conclusion

In this instalment of DX_ series, I shared how we upgraded our faculty career fair website's search function by shifting from basic front-end filtering to an advanced, type-ahead search with Typesense.

This cost-effective improvement was achieved by integrating Typesense's search engine and automating data indexing through a custom Node.js cron job.

The result was a significant enhancement in connecting students with employers, without altering the user interface, demonstrating the impact of open-source solutions and innovation in addressing project challenges and user requirements.

References

BONUS: Tech stack for the whole project

Frontend - React.js
Backend - Node.js
Database - MySQL
Search - Typesense
Image storage - Firebase storage
User tour - driver.js

Disclaimer

Although I contributed to the search feature, I was just one member of the whole project. The whole project was a successful effort of many undergraduates of the Faculty from different batches.
Code snippets and screenshots in this article may not be production-ready. They are edited as needed for educational purposes.

Part 3/3 - How to create a server-side timer using WebSockets (with Socket.IO), NestJS and Flutter

Rukshan J. Senanayaka — Mon, 15 Aug 2022 15:30:00 +0000

Introduction

In this part 3/3, I will show you how to implement WebSocket functionality in a new Flutter app and connect it to the NestJS backend server application that we implemented in part 2/3.

The code segments in this article are written for reading purpose, the full source code is available on my GitHub, linked at the end.

1. Create a new flutter project

I’m using Flutter 2.5.2 with null safety. A later version should support as well, but didn’t test with a newer version.

You can create a new Flutter project by executing following command.



flutter create server_timer_frontend

This can take a couple of minutes.

2. Install Socket.IO client package for Flutter

After your Flutter project is created, we can add the package of the socket client we are going to use.

I have tested several packages for this purpose. It was very difficulty to find a proper one because for our WebSocket connection to work successfully, both the backend Socket.IO version and the frontend client Socket.IO version must be compatible. So I had to do some trial and error and some online research to find out the best package.

The best package I found is named socket_io_client and its latest update (version 2.0.0) was released a few weeks ago (July of 2022). You can see more info at https://pub.dev/packages/socket_io_client.

You can install it by running



flutter pub add socket_io_client

One side note - the official Flutter documentation shows a way to implement WebSocket connection using a package called web_socket_channel, it was not working for me with my NestJS backend.

Before writing code, create a new folder called sockets within the lib folder, where we will put all of our code related to sockets. You can refactor them as however you like later.

3. Create socket service (socket_service.dart)

In the frontend, we have to have several files related to this implementation. One important file is socket_service.dart. Here we define methods such as connectAndListenToSocket(){}, disconnectSocket(){}, disposeSocket(){} which are important for our WebSocket communication.

The following is the abstract structure of the SocketService class.



class SocketService {
  static Socket? socket;

  // connectAndListenToSocket(){}

  // disconnectSocket(){}

  // disposeSocket(){}
}

First, write a method named connectAndListenToSocket(String authToken, String deviceId){}.

Next, within the connectAndListenToSocket(){} method, we have to set up the connection to the backend. Since the backend requires a user auth token, we have to send that as well with this.

Please edit the IP address according to your network connection. Otherwise this will not connect to the NestJS backend. You can view your IP from network settings of your computer.

The default port of the NestJS app is 3000. That is written in right side of the semicolon (:) as in the following code. (Replace the 192.168.X.X with your actual IP address)



    socket = io(
        'http://192.168.X.X:3000', // Replace with your network IP 
        OptionBuilder()
            .setTransports(['websocket'])
            .setAuth({
              'token': 'Bearer $authToken',
            })
            .setQuery({
              'deviceId': deviceId,
            })
            .disableAutoConnect()
            .build());

It takes two parameters, the user authToken (you can get this from Firebase Auth or any other auth method), the deviceId (you can get this by using device_info_plus package (https://pub.dev/packages/device_info_plus).

We can use the setQuery() method to send query parameters in the body of our message. This body can be caught by using the @MessageBody() decorator in NestJS.

The connectAndListenToSocket(){} method will try to connect the socket to backend.

Next, write the following piece of code which tries to actually start the WebSocket connection if it is not yet started.



    if (!socket!.connected) {
      socket!.connect();
      print('connecting....');
    }

Next, inside this connectAndListenToSocket(){} method, we can set up some listeners which are for main events such as onConnect(){}, onDisconnect(){}, onError(){} and onConnectError(){} etc. as below.



    socket!.onConnect((_) {
      print('connected and listening to socket!.');
    });

    socket!.onDisconnect((_) => print('disconnected from socket!.'));

    socket!.onError((data) => print(data));

    socket!.onConnectError((data) => {print(data)});

We can listen in the frontend, to the custom events set by us in the backend. Therefore, whenever the backend server sends a message under that event, this listener would be fired.

Next, write the following listener which listens to the tick event of our server-side timer app. We defined this event in the backend NestJS application.



    // When the message event 'tick' received from server, that data is added to a stream 'streamSocket'.
    socket!.on(TimerEvents.tick.toString().split('.').last, (data) {
      streamSocket.addResponse(data['timer'].toString());
    });

In that listener, we add the received data to a Flutter StreamController using the streamSocket.addResponse(data["timer"].toString()); method call. The variable data is the payload that we send from the backend. The value of the key timer contains the actual time in seconds.

Finally write the two functions for disconnecting the socket and disposing the socket.



  static disconnectSocket() async {
    socket!.disconnect();
  }

  static disposeSocket() async {
    socket!.dispose();
  }

The difference between the two is, disconnectSocket(){} does not remove the listeners created in our Flutter app for the WebSocket connection, but the disposeSocket(){} does.

If we connect to the same socket again, the multiple listeners would be present if we had only disconnected, and not disposed the WebSocket connection. Therefore, to start everything fresh next time, it's a good idea to disposeSocket(){} the socket when need to finish the WebSocket connection.

4. Create timer service (timer_service.dart)

In this service file, I will define the methods startServerTimer and stopServerTimer.



class TimerService {

  // startServerTimer(){}

  // stopServerTimer(){}

}

Whenever we want to send something to the backend, we use the socket.emit() method call. This method call would accept a first string parameter - the name of the event, and then the second parameter is the data to be sent as a JSON object.

We can accept this JSON object in our backend easily - using the @MessageBody() decorator in NestJS.



  // Timer methods.
  static startServerTimer(int duration) {
    if (SocketService.socket!.connected) {
      SocketService.socket!.emit(TimerEvents.timerStart.toString().split('.').last, {
        "dur": duration,
      });
    } else {
      print("No socket connection found.");
    }
  }

  static stopServerTimer() {
    if (SocketService.socket!.connected) {
      SocketService.socket!.emit(TimerEvents.timerStop.toString().split('.').last);
    } else {
      print("No socket connection found.");
    }
  }

5. Stream socket (stream_socket.dart)

This file is not essential for the socket implementation but I’m using it to store data received from the WebSocket connection and use it to update the UI easily using a StreamBuilder Flutter widget in my countdown timer example.



class StreamSocket {
  final StreamController _socketResponse = StreamController<String>.broadcast();
  Function(String) get addResponse => _socketResponse.sink.add;
  Stream<String> get getResponse => _socketResponse.stream.asBroadcastStream() as Stream<String>;
}

StreamSocket streamSocket = StreamSocket();

Of course there can be many other alternative ways to do this part. I would love to hear them in the comments below.

Don’t forget to import this file inside the socket_service.dart file.

6. Events file

In the flutter side also, I’m maintaining a socket_events.dart file to keep the needed socket events as enums, just like we did in NestJS in the part 2/3 of this tutorial series.



enum TimerEvents {
  tick,
  timerStart,
  timerStop,
}

Don’t forget to import this file inside the socket_service.dart file.

7. Add the UI to display socket data (main.dart)

The abstract structure of the main.dart is as follows.



void main() {
  runApp(const MyApp());
}

class MyApp extends StatelessWidget {

  // Setup MaterialApp

}

class MyHomePage extends StatefulWidget {

  // Initiate the state.

}

class _MyHomePageState extends State<MyHomePage> {

  // _convertToDisplayTime(){}

  // initState(){}

  // build(){}

}

First, clear out the comments from the default Flutter application.

Next, the important thing is, we have to write code inside the _MyHomePageState(){} class. Since the code is a bit long, I'll mention only the important things here, because you can get the full source code from the GitHub link at the end of the article.

We declare following function to convert the data from our backend to proper timer format for display.



  // This is to convert the time in seconds to a string with the format '00:00'.
  String _convertToDisplayTime(Duration duration) {
    String twoDigits(int n) => n.toString().padLeft(2, "0");
    String twoDigitMinutes = twoDigits(duration.inMinutes.remainder(60));
    String twoDigitSeconds = twoDigits(duration.inSeconds.remainder(60));
    if (duration.inHours == 0) {
      return "$twoDigitMinutes:$twoDigitSeconds";
    }
    return "${twoDigits(duration.inHours)}:$twoDigitMinutes:$twoDigitSeconds";
  }

When the WebSocket connection is established and connected, we are rendering a StreamBuilder widget on screen to display the data from that connection using the stream_socket.dart file we created earlier.



            (SocketService.socket != null && SocketService.socket!.connected)
                ? StreamBuilder(
                    stream: streamSocket.getResponse,
                    builder: (context, snapshot) {
                      if (snapshot.hasData) {
                        if (snapshot.data.toString() == "0") {
                          return const Text("Socket Status : TICKING TIMEOUT");
                        } else {
                          return Text(
                            'Socket Status : TIMER TICKING - ${_convertToDisplayTime(
                              Duration(
                                seconds: int.parse(snapshot.data.toString()),
                              ),
                            ).toString()}',
                          );
                        }
                      } else {
                        if (SocketService.socket!.connected) {
                          return const Text("Socket Status : CONNECTED");
                        } else {
                          return const Text("Socket Status : DISCONNECTED");
                        }
                      }
                    },
                  )
                : const Text('Socket Status : DISCONNECTED'),

Then we have to create the buttons which are there for the functions such as connect socket, disconnect, dispose, start timer and stop timer.

Here is the Connect button code.



            ElevatedButton.icon(
              onPressed: () {
                // Populate these two values with your own values.
                // Get token from Firebase Auth or other authentication service.
                // Get deviceId using `device_info_plus` package.
                SocketService.connectAndListenToSocket('token', 'deviceId'); // Start a socket channel with the server.

                // Update the UI when anything change in the socket.
                SocketService.socket!.onAny((event, data) {
                  setState(() {});
                });
              },
              label: const Text('Connect Socket'),
              icon: const Icon(Icons.connect_without_contact_rounded),
              style: ElevatedButton.styleFrom(
                primary: Colors.purple,
              ),
            ),

Here are the Start timer and Stop timer buttons code.



            ElevatedButton.icon(
              onPressed: () {
                TimerService.startServerTimer(10); // Start the server-side timer with 10 seconds.
              },
              label: const Text('Start Server Countdown'),
              icon: const Icon(Icons.timer),
              style: ElevatedButton.styleFrom(
                primary: Colors.green[900],
              ),
            ),
            ElevatedButton.icon(
              onPressed: () {
                TimerService.stopServerTimer();
              },
              label: const Text('Stop Server Countdown'),
              icon: const Icon(Icons.timer_off),
              style: ElevatedButton.styleFrom(
                primary: Colors.blue[900],
              ),
            ),

Here we are just calling the methods defined in TimerService to start a timer on the NestJS server or to stop it.

The disconnect socket does not remove the listeners, but dispose does. You can see the code for the relevant Disconnect socket and Dispose socket buttons below.



            ElevatedButton.icon(
              onPressed: () {
                // Disconnect from the socket. (Does not remove the listeners.)
                SocketService.disconnectSocket();
              },
              label: const Text('Disconnect Socket\n(Does not remove the listeners)'),
              icon: const Icon(Icons.remove_done),
              style: ElevatedButton.styleFrom(
                primary: Colors.purple,
              ),
            ),
            ElevatedButton.icon(
              onPressed: () {
                // Disconnect the socket and remove the listeners.
                SocketService.disposeSocket();
              },
              label: const Text('Dispose Socket\n(Fresh Start - removes all listeners)'),
              icon: const Icon(Icons.cancel),
              style: ElevatedButton.styleFrom(
                primary: Colors.purple,
              ),
            ),

That is all about the frontend UI part of the code!

Common errors

Error 1 ```

WebSocketException: Connection to 'http://192.168.X.X:3000/socket.io/?deviceId=deviceId&EIO=4&transport=websocket#' was not upgraded to websocket

Make sure you have correctly set the `SocketsGateway` as a **provider** in **app.module.ts** in the NestJS backend.

- Error 2

SocketException: OS Error: Network is unreachable, errno = 101, address = 192.168.X.X, port = XXXXX

Make sure you are connected to the correct network connection and that you are using the correct IP.

## Conclusion

That’s it. We have completed the flutter frontend implementation of WebSocket communication.

Hooray! 🥳🎉🎉🎉🥳🥳🥳🎉🥳🎉🎉 You've reached this far! Give yourself a pat on the back - you deserve it. 

I hope now you have some idea on how to implement a server-side timer using WebSockets(with Socket.IO), NestJS and Flutter. There may have been alternative ways to do some of the things that I have done in this tutorial series and I would love to hear about them in the comments section below.

Also, don't hesitate to point out any mistakes that I have made because I'm also learning this stuff! 

Please share your solution as well if you follow this series and make your own - I would love to see it. Have a wonderful day - Peace out ✌️

## Support me!

Do you think I deserve a cup of coffee for this article? 😃

<a href="https://www.buymeacoffee.com/rukshanjs" target="_blank"><img src="https://cdn.buymeacoffee.com/buttons/v2/default-yellow.png" alt="Buy Me A Coffee" style="height: 60px !important;width: 217px !important;" ></a>

## Video

[![IMAGE ALT TEXT HERE](https://img.youtube.com/vi/kx7LQNNmTX4/0.jpg)](https://www.youtube.com/watch?v=kx7LQNNmTX4)

## Source code

You can find the full source code of the project at my GitHub, https://github.com/RukshanJS/websockets-nestjs-flutter

## References

1. socket_io_client connection issue - https://stackoverflow.com/q/71679446

Part 2/3 - How to create a server-side timer using WebSockets (with Socket.IO), NestJS and Flutter

Rukshan J. Senanayaka — Mon, 15 Aug 2022 15:30:00 +0000

Introduction

In this part (part 2/3), I will discuss how to implement the backend server application using NestJS to establish a WebSocket connection.

The code segments in this article are written for reading purpose, the full source code is available on my GitHub, linked at the end.

NestJS is a wonderful backend framework to develop mid to large scale projects. It supports WebSockets officially and a very straightforward setup is available to get things going.

I am considering a single stateful server of NestJS and I will avoid Redis and microservices for now but they would be advisable to use for a fully-scalable application, to keep track of users connected to each WebSocket connection.

I mentioned this stateful because to keep track of all the connected users and their timers, a single variable called userTimers is maintained in our backend app.

That means if the server shuts down for some reason, that state would be lost and we would loose the timers for the users. If we use Redis and microservices knowledge, we could avoid this problem. But I think that should be for another topic.

Abstract architecture of our server-side timer app

The following diagram gives an abstract idea of the system we are going to implement. The method calls in the diagram are the ones usually used in a WebSocket connection. If you don't understand it fully, please go through this tutorial series again after reading all 3 parts for the first time. I have tried to simplify things as much as possible.

The things that we have to do for the backend implementation are the following steps.

1. Start a new NestJS project

Please install NestJS on your computer if you don't have already, by running the following command.



npm i -g @nestjs/cli

Then create a new NestJS project.



nest new server-timer-backend

2. Install the needed packages

The two packages that we require are provided by NestJS developers. Since I’ve tested this setup with NestJS version 8.0.0 (Although the latest version is 9), I’ll be using an earlier, supported version of the required packages. This setup is working as of early August, 2022.

@nestjs/websockets@8.4.7
@nestjs/platform-socket.io@8.4.7

We can install both of them by running the following command in the terminal.



npm i --save @nestjs/websockets@8.4.7 @nestjs/platform-socket.io@8.4.7

3. Create a sockets gateway (sockets.gateway.ts)

Firstly, I’ll create a folder named sockets where we will keep all of our files required for WebSocket communication for now. You can refactor these files in anyway you like.

Create a new file called sockets.gateway.ts.

Here the gateway file can be thought of as the main file in the backend related to WebSocket communication. It handles the connection, disconnection and any other message event.

This SocketsGateway class implements two interfaces which are provided by the @nestjs/websockets package.

We use the @WebSocketServer() decorator provided by the above package to tell NestJS to inject the WebSocket server.



@WebSocketGateway()
export class SocketsGateway
  implements OnGatewayConnection, OnGatewayDisconnect
{
  @WebSocketServer()
  public server: Server;

  // handleConnection(){}

  // handleDisconnect(){}

  // startMyTimer(){}

  // stopMyTimer(){}
}

I mentioned that an agreed-upon communication channel has to be created between client and server to start WebSocket communication. This agreement is established via a handshake.

Whenever this handshake is established, the handleConnection(){} method is invoked. This handshake is a real thing and its details can be accessed via client.handshake as a JSON with various information related to this WebSocket connection.



  handleConnection(@ConnectedSocket() client: any) {
    // console.log(
    //   `user ${client.user.id} with socket ${client.id} connected with device ${client.handshake?.query?.deviceId}`,
    // );

    client.join(
      getUserDeviceRoom(
        client.user.id,
        client.handshake.query.deviceId.toString(),
      ),
    );
  }

handleDisconnect(){} is more or less similar and it is invoked when an already connected socket is disconnected. You still have access to the handshake via client.handshake.



  handleDisconnect(@ConnectedSocket() client: any) {
    // console.log(
    //   `user ${client.user.id} with socket ${client.id} with device ${client.handshake?.query?.deviceId} DISCONNECTED`,
    // );

    client.leave(
      getUserDeviceRoom(
        client.user.id,
        client.handshake.query.deviceId.toString(),
      ),
    );
  }

A message event is a user-defined event to which the server listens (or subscribes). When a client (for example our flutter app) sends a message under that event, the server will execute whatever code is inside this listener method for that particular message event.

We are using two message events in our timer app - timerStart and timerStop.

In the startMyTimer(){} method, I use the @MessageBody() decorator to catch the payload sent from the client to server via the WebSocket connection. Here, the duration of the timer to be started in seconds, is contained in body.dur value. (We set this in our Flutter app)

In this example, I stop any existing timer for a user device before starting a new timer for that device.



  @SubscribeMessage(TimerEvents.timerStart.toString())
  startMyTimer(@ConnectedSocket() client: any, @MessageBody() body: any): void {
    // Stop any existing timer for this user device.
    stopTimerForUserDevice(
      client.user.id,
      client.handshake.query.deviceId.toString(),
    );

    // Start a new timer for this user device.
    startTimerForUserDevice(
      this.server,
      client.user.id,
      client.handshake.query.deviceId.toString(),
      body.dur, // Timer duration
    );
  }



  @SubscribeMessage(TimerEvents.timerStop.toString())
  stopMyTimer(@ConnectedSocket() client: any): void {
    // Stop current timer for this user device.
    stopTimerForUserDevice(
      client.user.id,
      client.handshake.query.deviceId.toString(),
    );
  }

4. Create a socket adapter (authenticated-socket.adapter.ts)

When we use WebSockets, we want to only allow authenticated users to connect with our backend WebSocket service. For that, we have to have a custom socket adapter (although one is given to us by default). This will make sure each and every request to establish a WebSocket connection will be made by a user authenticated in your backend.

First, create a class named AuthenticatedSocketAdapter which extends the IoAdapter from @nestjs/platform-socket.io package as below.



export class AuthenticatedSocketAdapter extends IoAdapter {
   // constructor(){}

   // createIOServer(){}
}

Next, write the constructor for this class. Here, if you have an AuthService implemented using Firebase Auth or any other auth service of your choice, just uncomment the commented lines shown in the code snippet below.

This authentication can be done using Firebase Auth or any other auth service of your choice and it doesn’t matter, as long as it will throw an exception for an unauthorized user. If the user is authorized, the WebSocket connection is established and if not, it is terminated by throwing Authentication error error.

If you don't have such an implementation yet, don't worry just leave these as is - we can test the implementation without authentication also.



  // private readonly authService:AuthService;
  constructor(private app: INestApplicationContext) {
    super(app);
    // this.authService = this.app.get(AuthService);
  }

We'll cover authentication in another tutorial so that the WebSocket connections will only be available to authenticated users.

Next, create a method named createIOServer(). Inside it, we are writing a middleware to grab the auth token from the handshake of the WebSocket connection, and authenticate that token using an AuthService (AuthService file is not implemented in this tutorial - if you don't have one yourself, you can ignore it for now because we don't need authentication for testing).

If the token is valid, the AuthService should return the relevant user from the authenticateToken() method. This returned user value is assigned to the socket.user variable. It is because we do this, we are able to access the relevant user as client.user by using the @ConnectedSocket() decorator in the sockets.gateway.ts file.

If the token is invalid, it will throw an error and the WebSocket connection would not be established.

Here also, if you have an AuthService implemented using Firebase Auth or any other auth service of your choice, just uncomment the commented lines shown in the code snippet below.

Please note that I have imported the Server type from socket.io library to our application as,



import { Server } from 'socket.io';

This is how I have imported Server type in sockets.gateway.ts, rooms.ts, events.ts files as well.



  createIOServer(port: number, options?: any) {
    const server: Server = super.createIOServer(port, options);

    server.use(async (socket: any, next) => {
      const tokenPayload: string = socket.handshake?.auth?.token;

      if (!tokenPayload) {
        return next(new Error('Token not provided'));
      }

      const [method, token] = tokenPayload.split(' ');

      if (method !== 'Bearer') {
        return next(
          new Error('Invalid authentication method. Only Bearer is supported.'),
        );
      }

      try {
        socket.user = {};
        // const user = await this.authService.authenticateToken(token);
        // socket.user = user;
        return next();
      } catch (error: any) {
        return next(new Error('Authentication error'));
      }
    });
    return server;
  }

5. Enable WebSockets in main.ts

In order to use the AuthenticatedSocketAdapter, the custom WebSocket adapter that we created, we should add the following code to main.ts file as below.



  const app = await NestFactory.create(AppModule);
  app.useWebSocketAdapter(new AuthenticatedSocketAdapter(app)); // Add our custom socket adapter.

Don’t forget to set the SocketsGateway as a provider in the app.module.ts



@Module({
  imports: [],
  controllers: [AppController],
  providers: [AppService, SocketsGateway],
})
export class AppModule {}

6. Create rooms.ts helper file

The rooms.ts is there to keep track of the connected users. This is the place where I will write the application logic related to our server-side timer. The methods in this file would send messages to clients as required.

First, create a global variable userTimers, to keep track of the connected users and their timers. Here, the keys of the object are the userId + deviceId combinations. The values are the timers of each user.



var userTimers = {}; // To keep track of timers for each user.

Here is how the userTimers variable would look like.



{
   "user001device001":Timeout {/* data of timer1 */},
   "user002device002":Timeout {/* data of timer2 */},
   "user003device003":Timeout {/* data of timer3 */}
}

But for more scalable applications, an approach which uses Redis and microservices and thereby making our backend application stateless would be advisable - then unexpected shutdowns of the backend server would not cause loss of user timers. This is because we can use Redis as a high speed key-value store.

Next, create a method getUserDeviceRoom(){}, to keep track of a single device of a user, based on userId and the deviceId. It returns the roomName as a string for that user device.



export function getUserDeviceRoom(userId: string, deviceId: string) {
  return `user:${userId}-device:${deviceId}`;
}

This way, we can use this returned roomName to create a conceptual room for that user device.

In the handleConnection(){} method, the connected client device join to this room. Then we can send a message to this specific device of a user, by sending that message to that room.

In the handleDisconnect(){} method, the connected client device leave from this room. We can no longer send messages to that user device until it establishes a socket connection again.

Next, create the method sendToUserDevice(){} to send the actual message using WebSockets.



export function sendToUserDevice(
  server: Server,
  userId: string,
  deviceId: string,
  event: string,
  payload: any,
) {
  server.to(getUserDeviceRoom(userId, deviceId)).emit(event, payload); // Actually send the message to the user device via WebSocket channel.
}

Next, we have to implement the methods to start the timer and stop the timer. I am using the setInterval() method of JavaScript to create a counter, and that counter value is sent to the user device in each tick.

At the same time, when a timer is created, I add it to a variable timer and it is stored in the variable userTimers that I mentioned in the beginning.



export function startTimerForUserDevice(
  server: Server,
  userId: string,
  deviceId: string,
  dur: number,
) {
  var counter = dur; // Set initial counter value to timer duration `dur` (in seconds).

  var timer = setInterval(function () {
    console.log(`counting ${counter}`);

    sendToUserDevice(server, userId, deviceId, TimerEvents.tick.toString(), {
      timer: counter,
    }); // Send tick message to user device.

    if (counter > 0) {
      counter--;
    } else {
      // Stop timer for this user.
      console.log(`user ${userId} has a timeout`);
    }
  }, 1000);

  userTimers[userId + deviceId] = timer; // Store timer for this user device.
}

We store the timers in the userTimers variable because we need a reference to the timer of each user, to stop it later if the client requests (using the Flutter app) to stop the server-side timer for that device. The relevant code is shown below.

After clearing the interval for that timer, we also delete it from the userTimers object, to release the memory allocated for it.



export function stopTimerForUserDevice(userId: string, deviceId: string) {
  clearInterval(userTimers[userId + deviceId]); // Stop the timer for this user device.

  delete userTimers[userId + deviceId]; // Delete the timer for this user device from the `userTimers` object.
}

Don’t forget to import this file inside the sockets.gateway.ts file.

7. Create events.ts helper file

The events.ts file just contains an enum to easily keep track of the message events I mentioned earlier.



export enum TimerEvents {
  tick = 'tick',
  timerStart = 'timerStart',
  timerStop = 'timerStop',
}

The reason to use enums is because this way we can use the variable name of the enum instead of typing the exact event name as a string everywhere that we need it - this reduces errors due to typing mistakes.

Don’t forget to import this file inside the sockets.gateway.ts file.

Conclusion

We wrote code in the following files in our NestJS server application to enable WebSocket communication.

sockets.gateway.ts
authenticated-sockets.adapter.ts
rooms.ts
events.ts
app.module.ts
main.ts

Although it is recommended to restructure these files according to a proper standard in a production application, this would be more than enough for us for now.

When you are done with this section, please move onto the next section of this 3-part series to see how to setup the flutter front end to connect to this backend WebSocket implementation.

Support me!

Do you think I deserve a cup of coffee for this article? 😃

Video

Source code

You can find the full source code of the project at my GitHub, https://github.com/RukshanJS/websockets-nestjs-flutter

References

Official NestJS docs on sockets -https://docs.nestjs.com/websockets/gateways
Why Socket.IO is needed? - https://stackoverflow.com/a/32811489/8995555
Why two packages are needed for WebSocket implementation in NestJS? - https://stackoverflow.com/a/73339808/8995555

Part 1/3 - How to create a server-side timer using WebSockets (with Socket.IO), NestJS and Flutter

Rukshan J. Senanayaka — Mon, 15 Aug 2022 15:30:00 +0000

If you have an interest in WebSockets technology, chances are that you have seen several tutorials online that use it to create applications such as realtime chat apps. But is that all that is possible with this WebSockets technology 🤔? - Of course not! We can do so much more with them.

In this 3-part tutorial series, we will learn how to create a server-side timer using NestJS and Flutter by application of WebSockets (with Socket.IO). You can use such an implementation to create a proper timer for a quiz app etc.

Pre-requisites

A decent knowledge on NestJS, Flutter, JavaScript/TypeScript.
Flutter 2.5.2 (with null safety), NestJS 8.0.0, Node 16.15.0, Android Emulator API S (This series of articles is successfully tested with these - may or may not work with other versions)

Final outcome of this 3-part article series

Introduction

This part (part 1/3) contains the basic theory and some other good-to-know info related to WebSocket communication.

In the world of connectivity, users need to send and receive data to and from various locations around the world. REST APIs are used widely to achieve this purpose. But in the traditional client-server communication using REST API and HTTP protocol, the server only responds to a request firstly made by the client - not initiate a data transfer by itself.

Therefore, to get new data, a client has to request new data each time it needs that updated data. But how does a client know whether it's the right time to request new data or to wait for some time until the data is updated in the server?

A few methods are available to achieve this,

Refresh the client to send a new HTTP request every time
Polling
WebSockets

Out of these, refreshing the client repeatedly is not good user experience. It is also not sometimes practical.

Polling is expensive in terms of computational power and resources required.

WebSockets is a high speed technology that can be used for realtime client-server communication.

The socket confusion

First off, let's clarify a couple of things.

There are a couple of jargon words used in socket communication (mostly interchangeably) in general, but it would be nice to have a clear idea about them. I'll just give a brief of some such words below and you can refer the references for more information on them.

WebSocket - is a protocol that depends on TCP protocol (like HTTP)
WebSockets - is the API specification of WebSocket protocol (supports two way communication)
Socket.IO - is a library (built on top of WebSocket protocol)
Socket - is an endpoint (in a socket communication)
Webhook - is an HTTP-based callback function which allows one way, server to server communication (not related to WebSockets and not concerned in this tutorial series)

Although Sockets can be divided as

TCP/IP sockets
Unix domain sockets,

we are only interested in TCP/IP sockets for this tutorial series.

Steps in establishing a WebSocket connection

Client sends an HTTP request to open a WebSocket connection (the HTTP connection will be upgraded to a WebSocket connection)
Server responds to the client’s request approving it with a 101 Switching Protocols response, and thereby completing the handshake and establishing a WebSocket connection.
This TCP/IP connection is kept open (by a ping-pong mechanism) allowing real time communication between the client and server until either the client or the server chooses to close the connection. This is a full-duplex communication, meaning both the client and server can send messages to each other at the same time.
After either side terminates the connection, the TCP resources would be unallocated.

In traditional client-server communication, a client sends a request to a server to retrieve some kind of a resource (a text, an image or an audio etc.) and waits for a response from the server. But in WebSocket communication, the server can send the resource when required - without a direct request from a client.

With the successful establishing of a handshake, a communication channel is created between client and server and then whenever the server wants to send something, it just can send it to the particular client without needing a REST API request from the client itself.

Either endpoint of this communication channel is called a Socket. Therefore, a socket has an IP address and a port number.

In this 3-part article series, we would be implementing a server-side timer using NestJS and Flutter, and we can start and stop it using our Flutter application.

Disclaimer

I'm also learning this stuff and there can be better ways to do the things that I have done in this tutorial series. Please feel free to give your feedback in the comments. If I have made a mistake, I'm more than grateful if you can bring them up in the comments section in each article 🤓

Conclusion

Sockets are a great way to implement realtime functionalities in applications. For example, WebSockets can be used to implement a server-side timer for a Flutter app.

This way, the clients have no way to tamper with or manipulate the timer on the frontend, and therefore this can be used, for example in quiz apps to maintain a server-side timer for each question with high integrity.

Now you can go to the part 2 of this tutorial series, to see how to implement the backend server application required for WebSocket communication, in NestJS.

Video

References

General concept of a socket - https://medium.com/swlh/understanding-socket-connections-in-computer-networking-bac304812b5c
WebSocket Protocol - https://datatracker.ietf.org/doc/id/draft-ietf-hybi-thewebsocketprotocol-09.html
WebSocket API (WebSockets) - https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API
Socket.IO can be more than just a wrapper for WebSockets - https://en.wikipedia.org/wiki/Socket.IO
WebSockets v Socket.IO - https://stackoverflow.com/a/62848079/8995555
WebSocket v Socket - https://stackoverflow.com/a/67826460/8995555
WebSockets v Webhooks - https://stackoverflow.com/a/24747947/8995555
Unix Socket v TCP/IP Socket (Just for additional knowledge) - https://www.baeldung.com/linux/unix-vs-tcp-ip-sockets
What is a Webhook - https://www.redhat.com/en/topics/automation/what-is-a-webhook
HTTP to WS upgrade request - https://stackoverflow.com/a/26401940/8995555

Introduction to Next.js - a personal opinion

Rukshan J. Senanayaka — Wed, 27 Jul 2022 18:23:00 +0000

Hi! First off, thanks for having a look at my very first article on dev.to 🥳! In this short article I’m going to give an introduction to Next.js while giving you my personal opinion on it as well. Your comments and ideas are also welcome 😎!

What’s in this article

What is Next.js
Some personal-favorite features of Next.js
Choosing Next.js v React

What’s not in this article

How to start a Next.js project

If you want to create only a very simple informational website, learning HTML and CSS is just enough. But if you want to add some functionality to that site, you may want to use JavaScript as well. Any website - no matter how complex, almost always can be simplified into HTML, CSS and JS code.

But what if you want to create a somewhat complex website or a web app instead of a simple one? That’s where web development libraries such as React.js and frameworks such as Next.js, Vue.js and Angular come in handy.

Next.js can be thought of as a full-stack web development framework, because it allows the developer to write code for frontend and backend in the same application. The frontend code is of course written in React using either JavaScript or TypeScript. (Although my personal favorite is with TypeScript - I’ll tell you why in a future article)

One major question I get asked is,

“Do I need to learn React to start learning Next.js?”

My answer to this question is,

“Don’t think of Next.js as a completely isolated thing from React, just think of it as an additional layer of help provided by the developers of Next.js to React developers because under the hood, Next.js is React!”

Check the following code snippets.

Hello World in React,

//src/App.jsx
import './App.css';

function App() {
 return (
   <div >
     Hello World
   </div>
 );
}

export default App;

Hello World in Next.js,

//pages/_app.tsx
import '../styles/globals.css';
import type { AppProps } from 'next/app';

function App({ Component, pageProps }: AppProps) {
 return <div>Hello World</div>;
}

export default App;

See, very similar. That’s because what Next.js does, is it takes care of some of the mundane yet common tasks needed to be done during the development of a React app.

As an example, developers who use React know the pain it takes to handle routes. If a route name is changed for example from auth/login to /login (among the many requests of real-world clients 😆) then a React developer has to manually change the route in the code. But, a Next.js developer only has to change the folder structure inside the pages folder. (This folder is a Next.js reserved folder for routing, by the way).

Features i like about Next.js

Page-based routing allows easy management of routing within the web app and it supports dynamic routes which allow easy handling of query params etc. If for example you have a projects/[projectId] sort of route, getting that projectId to display contents of a single project is very easy.
Hot code reloading which allows to see changes updated when saving a code file without server restart.
Easy deployment on various platforms like Vercel (offered by creators of Next.js) and Netlify.
Cool documentation which is very beginner-friendly. (But I was not able to get Server-side-rendering working yet by reading that).

What I don’t like about Next.js

Some third party packages which import global css in them will give an error. For example, https://github.com/uiwjs/react-md-editor/issues/52#issue-724437226. There are workarounds, but it’s an inconvenience.
Sometimes the dev version of the web app is too slow in the browser. No problem once built the production release.

Conclusion

Ok then bye bye React! Hello Next.js? Not so fast 😄! While Next.js is a very good framework for mid to complex projects, for very simple projects, it is wise to think of using plain React. (If it is even simpler than that, just use HTML, CSS and JS).

One final note - If it is a fairly complex client project, I almost always ask the client if it is okay to use Next.js or if they want pure React. If they have no preference, I always select Next.js because why not!

But don’t forget the good old HTML, CSS, JS because not all projects require React or Next.js. I mean who is to say you must use React for everything, right? I mean if you want to have a look at the most overengineered, and insanely complicated way to write a hello world web app, take a look at the video by Chris Hawkes - https://youtu.be/3nHQMAY_J1A, of course after giving me a like if you think I deserve one for this article 😃.

I'm also learning this stuff so let me know if I missed something in the comments below. Thanks and have a great day 🥳!

References

I was inspired by this article, please have a look at that also. - https://dev.to/olenadrugalya/introduction-to-nextjs-3gi4