DEV Community: Krishna Kurtakoti

Data for World Map Panel Plugin in Grafana from MySQL

Krishna Kurtakoti — Tue, 28 May 2024 10:09:07 +0000

Data for World Map Panel Plugin in Grafana:

Approach 1:

Using the InfluxDB data:

Here, the data is pulled from the database(InfluxDB) from a single measurement(table).

Schema of the hdb7 table is:

Time	DS_ID	from	graphStatement	latitude	longitude	to	totalPower	value

The above is the data that is pushed to the measurement(hdb7) of the InfluxDB database. Here, the hdb7 table’s graphStatement field values will be shown on the world map.

Query:

SELECT * from "hdb7" GROUP BY * ORDER BY DESC LIMIT 4

col1	col2
Location Data	table
Aggregation	current

Map Data Options

Field Mapping

col1	col2
Table Query Format	coordinates
Location Name Field	graphStatement
Metric Field	value
Latitude Field	latitude
Longitude Field	longitude

Field Mapping

DrawBack:

The requirement specifically mentions having 2 tables like shown below.
JOIN statements are not supported in InfluxDB.

Table1
DS_ID, Lat, Long

Table2
DS_ID, Sum(pump),sum(light),..total value

Table1:

DS_ID	latitude	longitude	value	time

Table2:

DS_ID	graphStatement	totalPower	value	time

The graphStatement covers the fields sum(boosterPump), sum(lighting), sum(lift) which are to be displayed on the map.

“JOIN is no longer a concept in 0.9. Series sharing a measurement can be queried together simply by omitting the differentiating tags from the WHERE clause.”
Link:
https://github.com/influxdata/influxdb/issues/624

However, there is one alternative called transformations in Grafana that can be used to join 2 tables of same datasource/mixed datasource as shown below.

Here, the data for Table1 (DS_ID, Lat, Long) will be queried from MySQL database. The data for Table2 (DS_ID, Sum(pump),sum(light),..total value) will be queried from InfluxDB database.

Implementation:

We can implement the above by choosing the datasource as Mixed.
The query 1 will be:
SELECT * from hdb10

The query 2 will be:
SELECT * from hdb9

The transform we are going to apply is :
Outer join: DS_ID

All the pictures are shown below:

When we change the metric field to Total(field from hdb10) we get the value on the map as shown below:

Approach 2:

Using the MYSQL data:

Here, the data is pulled from the database(MySQL) from a 2 tables.


    
// hidden setup JavaScript code goes in this preamble area
const hiddenVar = 42


    k@k-Lenovo-G50-70:~$  mysql -u root -p
Enter password: 
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 7

mysql> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| database_name      |
| mysql              |
| performance_schema |
| sys                |
+--------------------+
5 rows in set (0.00 sec)

mysql> use database_name
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed
mysql> show tables;
+-------------------------+
| Tables_in_database_name |
+-------------------------+
| a_one                   |
| total_power             |
| worldmap_latlng         |
| worldmap_latlng_a       |
+-------------------------+
4 rows in set (0.00 sec)

Table creation:
mysql>  INSERT INTO worldmap_latlng_a
    -> (lat,
    ->  lng,
    ->  DS_ID,
    -> value,
    -> timestamp)
    -> VALUES
    -> (1.3521,
    ->  103.8198,
    -> '0',
    -> 1.0,
    -> now());
Query OK, 1 row affected (0.13 sec)

mysql> CREATE TABLE worldmap_latlng_a (
    ->   id int(11) NOT NULL AUTO_INCREMENT,
    ->   lat FLOAT NOT NULL,
    ->   lng FLOAT NOT NULL,
    ->   DS_ID VARCHAR(20) NOT NULL,
    ->   value FLOAT NOT NULL,
    ->   timestamp timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
    ->   PRIMARY KEY (id)
    ->  ) AUTO_INCREMENT=7 DEFAULT CHARSET=latin1;
Query OK, 0 rows affected (0.65 sec)

mysql> describe total_power;
+-------------+-------------+------+-----+-------------------+-----------------------------+
| Field       | Type        | Null | Key | Default           | Extra                       |
+-------------+-------------+------+-----+-------------------+-----------------------------+
| id          | int(11)     | NO   | PRI | NULL              | auto_increment              |
| Lift        | float       | NO   |     | NULL              |                             |
| Lighting    | float       | NO   |     | NULL              |                             |
| Total       | float       | NO   |     | NULL              |                             |
| BoosterPump | float       | NO   |     | NULL              |                             |
| DS_ID       | varchar(20) | NO   |     | NULL              |                             |
| value       | float       | NO   |     | NULL              |                             |
| timestamp   | timestamp   | NO   |     | CURRENT_TIMESTAMP | on update CURRENT_TIMESTAMP |
+-------------+-------------+------+-----+-------------------+-----------------------------+
8 rows in set (0.00 sec)

mysql> describe worldmap_latlng_a;
+-----------+-------------+------+-----+-------------------+-----------------------------+
| Field     | Type        | Null | Key | Default           | Extra                       |
+-----------+-------------+------+-----+-------------------+-----------------------------+
| id        | int(11)     | NO   | PRI | NULL              | auto_increment              |
| lat       | float       | NO   |     | NULL              |                             |
| lng       | float       | NO   |     | NULL              |                             |
| DS_ID     | varchar(20) | NO   |     | NULL              |                             |
| value     | float       | NO   |     | NULL              |                             |
| timestamp | timestamp   | NO   |     | CURRENT_TIMESTAMP | on update CURRENT_TIMESTAMP |
+-----------+-------------+------+-----+-------------------+-----------------------------+
6 rows in set (0.00 sec)

Table1: worldmap_latlng_a
DS_ID
lat
lng
value
timestamp

Table2: total_power
DS_ID
Lift
Lighting
BoosterPump
Total
value
timestamp

INSERTING Records:

mysql>  INSERT INTO worldmap_latlng_a
    -> (lat,
    ->  lng,
    ->  DS_ID,
    -> value,
    -> timestamp)
    -> VALUES
    -> (1.3521,
    ->  103.8198,
    -> '0',
    -> 1.0,
    -> now());
Query OK, 1 row affected (0.13 sec)

mysql> INSERT INTO total_power
    -> (Lift,
    -> Lighting,
    -> Total,
    -> BoosterPump,
    -> DS_ID,
    -> value,
    -> timestamp)
    -> VALUES
    -> ( 10474.1997022,
    -> 8.97861111111,
    -> 14987.6236142,
    -> 4504.44530083,
    ->  '0',
    ->  1.0,
    ->  now());
Query OK, 1 row affected (0.11 sec)

DB Connection in Grafana(Add Data sources MySQL):
Host localhost:3306
Database database_name

SELECT CONCAT( "-") AS Conca, worldmap_latlng_a.lat, worldmap_latlng_a.lng, worldmap_latlng_a.DS_ID, total_power.Total
FROM total_power
INNER JOIN worldmap_latlng_a
ON worldmap_latlng_a.DS_ID = total_power.DS_I

The data we get is:
Conca lat lng DS_ID Total

1.4 104 0 14988

Here, we are getting both the data of the table1(worldmap_latlng_a) and the table2(total_power) from the join query executed above.

Database and model design - mongoose

Krishna Kurtakoti — Thu, 16 Dec 2021 05:13:31 +0000

A new user Alan Tang, on Stackoverflow posted a question regarding the top-level design of schemas for database.
The link to the question is here:
https://stackoverflow.com/questions/70345194/node-js-model-with-inner-nested-array/70345760#70345760
More detail information is below:

I have a question about my collection design.
Current design

Code (Part-I):


    
const customerSchema = mongoose.Schema({
customer_name: {
    type: String
},
purchase_history: [{
    amount: {
        type: Number,
        default: 0
    },
    currency: {
        type: String,
        require: true
    },
    description: {
        type: String
    }
}],
......
});

Every time, if a customer purchases a new item, it will push the history into the "purchase_history".
The purpose of "purchase_history" is to let them check their own history.
Is this a good idea? or If you have a good idea, please few free to share.
Thank you

I answered to the above question like this:
I have created 2 different schemas, one for the customer and another for the purchase. Please follow the standard procedure to keep all the service files, model files, controller files in separate folders.
The below is the customer model.

customer.model.js
Code (Part-II):


    
const mongoose = require('mongoose');
let Schema = mongoose.Schema;

const CustomerSchema = mongoose.Schema({
  customer_name: {
    type: String,
  },
});

const customer = mongoose.model('customer', CustomerSchema);
module.exports = customer;

We have the purchase model as:

purchase.model.js
Code (Part-III):


    
const mongoose = require('mongoose');
let Schema = mongoose.Schema;
const customer = require('./customer.model');

var purchaseSchema = new Schema(
  {
    customerId: { type: Schema.Types.ObjectId, ref: 'customer' },
    amount: {
      type: Number,
      default: 0,
    },
    currency: {
      type: String,
      required: true,
    },
    description: {
      type: String,
    },
  },
  { timestamps: true }
);

module.exports = mongoose.model('purchase', purchaseSchema);

Here, we can see the customer data is stored in customer collection and the purchase data stored in the purchase collection. Each purchase record has a reference field 'customerId' which is the customer's unique identitifer. This field is defined in the purchase model. The customer's purchase history can be fetched by quering for the customerId field.
We can create an api for fetching customer's purchases as:

purchase.service.js
Code (Part-IV):


    
const purchaseModel = require('./purchase.model');

module.exports.getByCustomerId = async (_customerId) => {
  try {
    const purchaseList = await purchaseModel.find({
      customerId: _customerId,
    });
    return purchaseList;
  } catch (err) {
    throw err.message;
  }
};

DESIGN PRINCIPLE:
The design principle followed here is to avoid duplication as suggested by a Senior developer. It is not good practice to store the same value in different collections, the purchase data being stored in the customer collection as in Alan’s schema.

Designing backend for performance by reducing DB calls.

Krishna Kurtakoti — Mon, 13 Dec 2021 06:50:01 +0000

We need to generate report based on the user’s activities for each day. A wallet for each user is created at the time of user registration. The user can top up his wallet and use his wallet to make purchases by shopping in the application’s platform. All these activities are accounted in the reports generated daily.
Code (Part-I):


    
const userWalletService = require("../../services/shop/userWallet");

let from = reportGenerationStartDate(dd/mm/yyyy00:00:00)
let to = reportGenerationEndDate(dd/mm/yyyy23:59:59)

for(let user of users){
////fetch the user’s activities from the DB(database) 

const filter = {
    updatedAt: {
      $gte: from,
      $lt: to,
    },
   userId: userId
  };
  const projection = {};
  const option = {};
  const userWallet = await userWalletService.getAllUserWallets({
    filter,
    projection,
    option,
  });

logger.info(
    Got all userWallet in ${dateRange} = ${userWallet}
  );
 openingBalance = userWallet.walletBalance;
}

Here, we are fetching user’s wallet data based on the user’s Id. For each user, we are making a DB call. If the users count increases as the application usage increases, say to 1000 or 10000 users, we need to make 10000 calls to the DB to fetch each user’s information and use that to generate the report. This is not feasible or good way to fetch the data from database.

From the above code, we can see that DB calls inside for loop can make the database to become unresponsive (in our case MongoDB to send error – pool draining and close the connection without processing the current DB calls being made).

This is undesirable and must be avoided. The approach to follow as suggested by one of the Project Lead is to fetch all the data that would be fetched when each call would be made separately inside the for loop.
One we have the data, we can do the processing, filtering operations with the modern Javascript functions like filter(), find(), map(), etc.

In any application, the DB calls can always be treated as bottle neck for processing the Api requests.
Also, if the app is running in cloud, we must know the processing power will be huge. Any large volumes of data is handled efficiently by these apps running on cloud due to automatic scaling of the resources when required.
We have our application running on AWS cloud and the framework is serverless with lambda functions which have high processing power, can do computing operations on large set of data efficiently.

We need to always consider DB calls made as having constraints on the database resources and should limit the number of calls to the database.

The above code is modified as shown:

We declare an array containing all the userIds based on which the user’s activities can be fetched.
We make a query like shown below.
userId: { $in: userIds }

Code (Part-II):


    
const userWalletService = require("../../services/shop/userWallet");

const filter = {
    updatedAt {
      $gte: from,
      $lt: to,
    },
   userId: { $in: userIds }
  };
  const projection = {};
  const option = {};

const userWallets = await userWalletService.getAllUserWallets({
    filter,
    projection,
    option,
  });

logger.info(
    Got all userWallets in ${dateRange} = ${userWallets.length}
  );

for (let i = 0; i < users.length; i++) {
const userWallet = userWallets.find((userWallet) => {
return userWallet.userId.toString() == users[i]._id.toString();
});
openingBalance = userWallet.walletBalance;
}

Grafana Dashboard:

Krishna Kurtakoti — Sat, 13 Feb 2021 06:13:34 +0000

In this post, we will see how to use different visualizations, like the simple graph, pie chart, world map panel in the grafana dashboard by writing queries in Influx query language and fetching data from the InfluxDB database which is a time-series database.
a. Data Visualization
b. Installing Plugins Docker Grafana
c. Writing to the InfluxDB2.0

LINE CHARTS:

1.
As we can see, we are showing the records for 2 different blocks, i.e, DS_Id = 0 and DS_Id = 1.The timestamp is the same for both the blocks, 1598918400, 1598918402,
1598918403 ….. .The date for the above can be obtained from the link: https://www.epochconverter.com/

TIMESTAMP	POWER_A	POWER_B	POWER_C
1598918400	1403.421	712.372	1680.471
1598918402	1423.817	731.249	1680.658
1598918403	1444.172	749.339	1700.859
1598918404	1774.402	1106.427	2041.954
1598918405	1774.402	1106.427	2041.954

TIMESTAMP	DS_Id	POWER_A	POWER_B	POWER_C
1598918400	1	821.847	574.748	1203.807
1598918402	1	823.367	574.315	1203.795
1598918403	1	819.939	574.261	1203.647
1598918404	1	819.939	574.261	1203.647

Our requirement is to get the aggregated power for POWER_A, POWER_B, POWER_C fields.
For example, for the timestamp 1598918400,

TIMESTAMP	DS_Id	POWER_A	POWER_B	POWER_C
1598918400	0	1403.421	712.372	1680.471
1598918400	1	821.847	574.748	1203.807

we have values for POWER_A as 1403.421W and 821.847W, sum as 2225.268W. Likewise, we have to calculate for all the time-series values(1598918402, 1598918403, 1598918404, …) for POWER_A, POWER_B and POWER_C also.

This computation has to be done by Grafana. We are implementing this using the SQL syntax like queries as shown below:

We are going to compute the aggregated power for the field POWER_A now.
All the queries were constructed and executed successfully by Narendra Reddy Mallidi, SQL Developer.
We have our first query below:
SELECT POWER_A as AA FROM "hdb2" WHERE "DS_ID" = '0'
Here, POWER_A is the variable where are going to fetch from the table(table is called measurement in InfluxDB queries) named "hdb2".

The same thing is followed for other 4 blocks also(DS_ID =’1’, DS_ID=’2’, DS_ID=’3’, DS_ID=’4’)

SELECT POWER_A as BB FROM "hdb2" WHERE "DS_ID" = '1'

SELECT POWER_A as CC FROM "hdb2" WHERE "DS_ID" = '2'

SELECT POWER_A as DD FROM "hdb2" WHERE "DS_ID" = '3'

SELECT POWER_A as EE FROM "hdb2" WHERE "DS_ID" = '4'

Now, we compute the aggregated power for POWER_A with the below query:

SELECT sum(ds1) + sum(ds0) + sum(ds2) + sum(ds3) + sum(ds4) AS Total
FROM
(
SELECT POWER_A as ds0 FROM "hdb2" WHERE "DS_ID" = '0'
),
(
SELECT POWER_A as ds1 FROM "hdb2" WHERE "DS_ID" = '1'
),
(
SELECT POWER_A as ds2 FROM "hdb2" WHERE "DS_ID" = '2'
),
(
SELECT POWER_A as ds3 FROM "hdb2" WHERE "DS_ID" = '3'
),
(
SELECT POWER_A as ds4 FROM "hdb2" WHERE "DS_ID" = '4'
) WHERE $timeFilter GROUP BY time(1s) fill(null)

Here, hdb2 is the table name in our INFLUXDB1.8 database from where we are fetching our data into the Grafana dashboard running at port 3000.

We get the graph as shown below:

TIMESTAMP	SAMPLE	POWER_A
2020-09-01 08:27:16	Sample 1	7.92k

TIMESTAMP	SAMPLE	POWER_A
2020-09-01 08:17:08	Sample 2	5.77k

(Sample 2):
TIMESTAMP: 2020-09-01 08:17:08
Aggregate_Power: 5.77k

We are getting the correct aggregated values for POWER_A. We need to do the same for POWER_B and POWER_C.

For POWER_B:

SELECT sum(ds1) + sum(ds0) + sum(ds2) + sum(ds3) + sum(ds4) AS PB
FROM
(
SELECT POWER_B as ds1 FROM "hdb2" WHERE "DS_ID" = '1'
),
(
SELECT POWER_B as ds0 FROM "hdb2" WHERE "DS_ID" = '0'
),
(
SELECT POWER_B as ds2 FROM "hdb2" WHERE "DS_ID" = '2'
),
(
SELECT POWER_B as ds3 FROM "hdb2" WHERE "DS_ID" = '3'
),
(
SELECT POWER_B as ds4 FROM "hdb2" WHERE "DS_ID" = '4'
) WHERE $timeFilter GROUP BY time(1s) fill(null)

For POWER_C:

SELECT sum(ds1) + sum(ds0) + sum(ds2) + sum(ds3) + sum(ds4) AS PC
FROM
(
SELECT POWER_C as ds1 FROM "hdb2" WHERE "DS_ID" = '1'
),
(
SELECT POWER_C as ds0 FROM "hdb2" WHERE "DS_ID" = '0'
),
(
SELECT POWER_C as ds2 FROM "hdb2" WHERE "DS_ID" = '2'
),
(
SELECT POWER_C as ds3 FROM "hdb2" WHERE "DS_ID" = '3'
),
(
SELECT POWER_C as ds4 FROM "hdb2" WHERE "DS_ID" = '4'
) WHERE $timeFilter GROUP BY time(1s) fill(null)

We will get the graph as shown below:

TIMESTAMP	SAMPLE	POWER_A
2020-09-01 08:27:12	Sample 3	7.89k

(Sample 3):
TIMESTAMP: 2020-09-01 08:27:12
Aggregate_Power: 7.89k

TIMESTAMP	SAMPLE	POWER_A
2020-09-01 08:17:08	Sample 4	5.77k

(Sample 4):
TIMESTAMP: 2020-09-01 08:17:08
Aggregate_Power: 5.77k

As we can see, these aggregated values for POWER_A(sample 3 and 4) confirm with the values shown in the previous graphs (sample 1 and 2) with the same timestamp.

PIE CHART VISUALIZATION:
Below is the query for visualization of 3 components:
Lift_Energy, Public_Lighting_Energy, Booster_Pump_Energy.
**
SELECT
(( (sum("LIFT_TOTAL")) / (($to - $from) / 1000) ) )* (($to - $from) / 3600000)
as Lift_Energy,
(((sum("LIGHT_TOTAL")) / (($to - $from) / 1000))) * ((($to - $from) / 1000) / 3600)
as Public_Lighting_Energy,
(((sum("PUMP_TOTAL")) / (($to - $from) / 1000))) * ((($to - $from) / 1000) / 3600) as Booster_Pump_Energy
FROM "hdb2" WHERE $timeFilter GROUP BY time(24h)
**
Here, we have combined 3 subqueries.

(sum("LIFT_TOTAL")) / (($to - $from) / 1000) ) )* (($to - $from) / 3600000) as Lift_Energy
(sum("LIGHT_TOTAL")) / (($to - $from) / 1000))) * ((($to - $from) / 1000) / 3600) as Public_Lighting_Energy
(sum("PUMP_TOTAL")) / (($to - $from) / 1000))) * ((($to - $from) / 1000) / 3600) as Booster_Pump_Energy All the 3 queries are similar. We will take 1st query. The records are taken for each second,i.e, for every consecutive second we are recording the data. Here, (sum("LIFT_TOTAL")) is the sum computed over the period mentioned - (($to - $from) in the time window.

The query has been updated by Grafana as:
SELECT
(( (sum("LIFT_TOTAL")) / ((1599190800000 - 1598898600000) / 1000) ) )* ((1599190800000 - 1598898600000) / 3600000)
as Lift_Energy,
(((sum("LIGHT_TOTAL")) / ((1599190800000 - 1598898600000) / 1000))) * (((1599190800000 - 1598898600000) / 1000) / 3600)
as Public_Lighting_Energy,
(((sum("PUMP_TOTAL")) / ((1599190800000 - 1598898600000) / 1000))) * (((1599190800000 - 1598898600000) / 1000) / 3600) as Booster_Pump_Energy
FROM "hdb2" WHERE time >= 1598898600000ms and time <= 1599190800000ms GROUP BY time(24h)

Since the precision is in milliseconds, we are dividing it by 1000.Now, we get total power for the time range applied. The unit for energy consumption is watt-hour(Wh). Example, a 40-watt electric appliance operating continuously for 25 hours uses one kilowatt-hour. The value ((($to - $from) / 1000) / 3600) gives the total operating hours which multiplied with the total power,i.e,(sum("PUMP_TOTAL")) / (($to - $from) / 1000))) gives power consumption in watt-hour units.

WORLD MAP VISUALIZATION:
The sample csv table we are using for the world map is shown below.
| Time | DS_ID | graphStatement | latitude | longitude | value | totalPower |
| ------------|:-----:| -----------------------------------: | -------: | --------: |
----- |:----------:|
| 2020-12-27 16:02:27 | 4 | Total: 49836, lift: 18377, light: 3357, pump: 54699339, DS_ID: 4 | 1.3 | 104 | 1 | 49836 |

Above is the csv file which we are uploading in the influxdb into the "hdb7" measurement.
Query for world map:
SELECT * from "hdb7" GROUP BY * ORDER BY DESC LIMIT 4
We can see the query and the settings also in the image below. Here, we are using latitude and longitude values to plot the map and put the graphStatement field in the label of the map.

b. More on Installing Plugins Docker Grafana:

Open the SSH terminal on your machine and run the following command:

ssh your_username@host_ip_address

If the username on your local machine matches the one on the server you are trying to connect to, you can just type:

ssh host_ip_address
And hit Enter.
After successful login, execute the below commands in the shell:
sudo docker ps -a
sudo docker exec -it --user=root grafana /bin/sh
grafana-cli plugins install grafana-worldmap-panel
sudo docker container stop d1ead747ec87
sudo docker start d1ead747ec87
1.
The sudo docker ps -a lists all the running containers.
2.
We can execute/test the commands for the application running inside the container with sudo docker exec -it --user=root grafana /bin/sh command.
We can also ping to test the port connections with the commands:
curl http://localhost:3000
curl http://160.100.100.204:3000 where the ip-address is of the remote virtual machine provisioned in the cloud which we have logged into.
3.
The plugins are installed with grafana-cli plugins install grafana-worldmap-panel.
4.
The docker conatiners are restarted with commands:
sudo docker container stop d1ead747ec87
sudo docker start d1ead747ec87


    
// hidden setup JavaScript code goes in this preamble area
const hiddenVar = 42


    
Last login: Fri Dec 18 19:37:21 2020 from 49.206.11.161
root@d4eViz:~# sudo docker ps -a
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS                    NAMES
d1ead747ec87        grafana/grafana     "/run.sh"           46 hours ago        Up 17 hours         0.0.0.0:3000->3000/tcp   grafana
root@d4eViz:~# sudo docker exec -it --user=root grafana /bin/sh
/usr/share/grafana # grafana-cli plugins install grafana-worldmap-panel
installing grafana-worldmap-panel @ 0.3.2
from: https://grafana.com/api/plugins/grafana-worldmap-panel/versions/0.3.2/download
into: /var/lib/grafana/plugins

✔ Installed grafana-worldmap-panel successfully 

Restart grafana after installing plugins . 

/usr/share/grafana # sudo docker container stop d1ead747ec87
/bin/sh: sudo: not found
/usr/share/grafana # root@d4eViz:~# 
root@d4eViz:~# sudo docker container stop d1ead747ec87
d1ead747ec87
root@d4eViz:~# sudo docker start d1ead747ec87
d1ead747ec87
root@d4eViz:~#


    
// hidden setup JavaScript code goes in this preamble area
const hiddenVar = 42


    
root@d4eViz:~# sudo docker rm -fv $(sudo docker ps -aq)
08d6b4e38932
root@d4eViz:~# sudo docker run -d -p 3000:3000 --name=grafana -v grafana-storage:/var/lib/grafana grafana/grafana
d1ead747ec87a566c5f8de5c36a705d3b8e1860f7e7dc78b2ea5bf2ef0f574d8
root@d4eViz:~# sudo docker ps -a
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS                    NAMES
d1ead747ec87        grafana/grafana     "/run.sh"           4 seconds ago       Up 4 seconds        0.0.0.0:3000->3000/tcp   grafana
root@d4eViz:~# sudo docker exec -it grafana /bin/sh
/usr/share/grafana $ curl http://localhost:3000
/bin/sh: curl: not found
/usr/share/grafana $ apk add curl
ERROR: Unable to lock database: Permission denied
ERROR: Failed to open apk database: Permission denied
/usr/share/grafana $ root@d4eViz:~# 
root@d4eViz:~# sudo docker exec -it --user=root grafana /bin/sh
/usr/share/grafana # apk add curl
fetch http://dl-cdn.alpinelinux.org/alpine/v3.12/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.12/community/x86_64/APKINDEX.tar.gz
(1/3) Installing nghttp2-libs (1.41.0-r0)
(2/3) Installing libcurl (7.69.1-r3)
(3/3) Installing curl (7.69.1-r3)
Executing busybox-1.31.1-r19.trigger
Executing glibc-bin-2.30-r0.trigger
/usr/glibc-compat/sbin/ldconfig: /usr/glibc-compat/lib/ld-linux-x86-64.so.2 is not a symbolic link

OK: 27 MiB in 37 packages
/usr/share/grafana # curl http://localhost:3000
Found.

/usr/share/grafana # curl http://160.100.100.204:3000
Found.

/usr/share/grafana #

c. Writing to the InfluxDB2.0
Reference:

https://john.soban.ski/refactor-python-to-influx-2.html

Requirements:

Get started with InfluxDB 2.0

The InfluxDB 2.0 time series platform is purpose-built to collect, store, process and visualize metrics and events. Get started with InfluxDB OSS v2.0 by downloading InfluxDB, installing the necessary executables, and running the initial setup process.

If not installed, follow the link
https://docs.influxdata.com/influxdb/v2.0/get-started/

Start InfluxDB by running the influxd daemon:

k@k-Lenovo-G50-70:~$ influxd

Python installed

k@k-Lenovo-G50-70:~$ python --version
Python 2.7.12
k@k-Lenovo-G50-70:~$ python3 --version
Python 3.5.2

Steps:

Replace the values of INFLUX_TOKEN, ORG, BUCKET_NAME and measurement_name with the name of the table you need to create.

Also replace the csv path you need to upload the csv file at line:
with open('/home/k/Downloads/influxData/data_0_20200901.csv') as csv_file:
In the csv file, we have time stored in Unix Timestamp format.
Run the below program:

k@k-Lenovo-G50-70:~/a_b$ python3 pushToInflux2_0.py

/home/k/a_b/pushToInflux2_0.py

Code:


    
// hidden setup JavaScript code goes in this preamble area
const hiddenVar = 42


    
'''
!/usr/bin/python
'''
import requests
import uuid
import random
import time
import sys
import csv
import json

INFLUX_TOKEN='qCAYOyvOErIP_KaJssk_neFar-o7PdvHL64eWYCD_ofywR_J3iubktdB58A3TE-6sM7C61Gt8qOUPvc4t0WVBg=='
ORG="asz"
INFLUX_CLOUD_URL='localhost'
BUCKET_NAME='b'
'''
Be sure to set precision to ms, not s
'''
QUERY_URI='http://{}:8086/api/v2/write?org={}&bucket={}&precision=ms'.format(INFLUX_CLOUD_URL,ORG,BUCKET_NAME)
headers = {}
headers['Authorization'] = 'Token {}'.format(INFLUX_TOKEN)
measurement_name = 'data_0_20200901'
'''
Increase the points, 2, 10 etc.
'''
number_of_points = 1000
batch_size = 1000

with open('/home/k/Downloads/influxData/data_0_20200901.csv') as csv_file:
    csv_reader = csv.reader(csv_file, delimiter=',')
    print('Processed')
for row in csv_reader:
    _row = 0
    if row[0] == "TIMESTAMP":
        pass
    else:
        _row = int((int(row[0])) * 1000)
    print(_data_end_time, row[0],_row, '\n')
    data.append("{measurement},location={location} POWER_A={POWER_A},POWER_B={POWER_B},POWER_C={POWER_C} {timestamp}"
        .format(measurement=measurement_name, location="reservoir", POWER_A=row[2], POWER_B=row[3], POWER_C=row[4], timestamp=_row))

count = 0
if name == 'main':
  # Check to see if number of points factors into batch size
  count = 0
  if (  number_of_points % batch_size != 0 ):
    raise SystemExit( 'Number of points must be divisible by batch size' )
  # Newline delimit the data
  for batch in range(0, len(data), batch_size):
    time.sleep(10)
    current_batch = '\n'.join( data[batch:batch + batch_size] )
    print(current_batch)
    r = requests.post(QUERY_URI, data=current_batch, headers=headers)
    count = count + 1
    print(r.status_code, count, data[count])

In the InfluxDB2.0 screen page at: http://localhost:8086/orgs/772821f18f2f7a0d/data-explorer?bucket=b Under the Explore option

We need to set the WINDOW PERIOD as 15s / 1m to see more points on the dashboard.
Also, the timing window must be adjusted according to the timestamp of the record.
For e.g,

TIMESTAMP	DS_Id	POWER_A	POWER_B	POWER_C
1598918400	0	1403.421	712.372	1680.471

So, we need to apply the time range from the above date for the results to show as in the above window.

My Github Profile for code:
Please see the master branch in my repo.
[Link]https://github.com/krishnakurtakoti/python-influxdb-2.0-write

Azure Devops Series - 2

Krishna Kurtakoti — Sat, 13 Feb 2021 06:13:24 +0000

Creating Release Pipelines
Pre-requisites:

We need to have a service connection ready to bind the azure web app with the devops pipelienes. So, we proceed by creating a new service connection as shown below:

In the project Settings, search for Service connections.

a. Click on Service connections as shown in the figure below. Select Azure Resource Manager as connection type.

b. For the selected Azure Resource Manager, select Publish Profile as the Authentication method.

c. The final step of creating a new Azure service connections is completed by filling in the Subscription, Resource group(where the web app is deployed), the web app name, Service connection name.

Now, the service connection is created.

Create a new release pipeline

a. Create a new pipeline as shown in the figure below by clicking ‘new release pipeline’.

b. Create a new task by searching for ‘Azure Web App’ under the Deploy section.

c. Click on Azure Web App as shown in the figure below.

d. Fill in the Display name, Azure subscription,App type, App name

e. We also need to add an artifact produced earlier from the build pipeline as shown below.

f. Once, the artifact is uploaded, we need to check the continous integration checkbox for us to run the pipeline whenever we have a new build pipeline running. It is shown below.
Click on add to specify the branch to build.Here, we have ‘Development’ as the branch.

g. Final step is to click on Create release button to trigger the continous deployment of the release pipeline.

We can see the release pipeline running as shown below.

This completed the flow for building of the release pipleline by creating a new service connection in the Azure Devops in our project.

I would like to mention all who helped me in development of the project.

Althaf H(CTO) in building the project flow and reviewing it.
Swetha SR (.Net Solution Architect) for developing the build pipelines (translating from manual build cycle to automated build pipelines).
Narendra Reddy(.Net developer) in debugging the project.

Azure Devops Series - 1

Krishna Kurtakoti — Sat, 13 Feb 2021 06:13:16 +0000

Steps for import project to Azure Repos from Git:

Create a new project.
After getting the ‘Welcome to the project!’ screen, click on the repos.
Under the Import a repository section, click on the import button.
Under the Import a Git repository screen as shown below, enter your git link, credentials as shown below:
After clicking import button, we get the screen as shown below:
Repository type: Git

Clone URL *: https://repo-url.git.beanstackapp.com/project-workshift.git

Username:

Password / PAT *:

After successful import, we get all the code into the repos folder along with all the commits and the branches as shown below:

This completes successful import of the git repository into the Azure Repos. This repository in Azure Repos can be used for Continous Integration and Continous Delivery by creating Build pipelines and Release pipelines as shown in the next post of the series.

Otp genertion and verification using speakeasy, Nest.js and MongoDB

Krishna Kurtakoti — Wed, 11 Nov 2020 07:09:41 +0000

Here, we have designed and developed a flow for OTP(One time password) for user registration and also blocking a user's account after the maximum retries for incorrect otp is exceeded.
We will go in steps for generation, verification and blocking a user's account.

Step 1:
OTP Generation:
Code (Part-I):
src/auth/auth.controller.ts


    
// hidden setup JavaScript code goes in this preamble area
const hiddenVar = 42


    
import {
  Controller,
  Post,
  Req,
  UseGuards,
  Get,
  Body,
  BadRequestException,
  Param,
  NotFoundException,
} from "@nestjs/common";
import { JwtAuthGuard } from "./auth.guard";
import { LoggedInToken } from "../users/objects/login-user.dto";
import { AuthService } from "./auth.service";
import * as speakeasy from "speakeasy";
import { optSecret } from "../common/constants/config";
import {
  UNKNOWN_PARAM,
  EMAIL_NOT_FOUND,
  OTP_ERROR,
  EXISTS,
  OTP_NOT_EXPIRED,
  NEW_PASSWORD_AND_CONFIRM_NEW_PASSWORD_ERROR,
  OTP_TIME_OUT,
  TOKEN_ALREADY_USED,
  EMAIL_ERROR,
  BLOCKED_ACCOUNT_ERROR,
} from "../common/constants/string";
import { plainToClass } from "class-transformer";
import { success } from "../common/base/httpResponse.interface";
import { UserDto } from "../users/objects/create-user.dto";
import { OtpEmail, UserCycloanAccountBlockedEmail } from "../users/objects/user.registered.email";
import {
  ForgetPasswordOtpEmail,
  PasswordChangedAlert,
} from "../users/objects/user.registered.email";
import { EmailService } from "../email/email.service";
import { OtpService } from "./otp/otp.service";
import { RequestUser } from "../common/utils/controller.decorator";
import { UsersService } from "../users/users.service";
import { EmailDto } from "../email/objects/email.dto";
import { OtpDto } from "./otp/otp.dto";
import { InjectModel } from "@nestjs/mongoose";
import { IOtp, Otp } from "./otp/otp.schema";
import { Model } from "mongoose";
import { ForgotPasswordOtpService } from "./forgot-password-otp/forgot-password-otp.service";
import { ForgotPasswordOtp } from "./forgot-password-otp/forgot-password-otp.schema";
import { ForgotPasswordOtpDto } from "./forgot-password-otp/forgot-password-otp.dto";
import { OtpIncorrectService } from "./otpIncorrect/otpIncorrect.service";
import { OtpIncorrect } from "./otpIncorrect/otpIncorrect.schema";
import { BlockedAccountService } from "./blockedAccounts/blockedAccounts.service";
import { IBlockedAccount } from "./blockedAccounts/blockedAccounts.schema";
import { OTP_RETRY_LIMIT, Status, ROLES_ACCESS_ACTION, BLOCKED_ACCOUNT_TYPE } from "../common/constants/enum";
import { RolesService } from "../roles/roles.service";
import { OtpIncorrectForgotPasswordService } from "./otpIncorrectForgotPassword/otpIncorrectForgotPassword.service";
import { OtpIncorrectForgotPassword } from "./otpIncorrectForgotPassword/otpIncorrectForgotPassword.schema";

//@UseGuards(JwtAuthGuard)
@Controller("auth/refresh")
export class AuthController {
  constructor(
    private authService: AuthService,
    private emailService: EmailService,
    private usersService: UsersService,
    private otpService: OtpService,
    private forgotPasswordOtpService: ForgotPasswordOtpService,
    @InjectModel("Otp") private readonly otpModel: Model,
    @InjectModel("ForgotPasswordOtp")
    private readonly forgotPasswordotpModel: Model,
    private readonly otpIncorrectService: OtpIncorrectService,
    @InjectModel("OtpIncorrect") private readonly otpIncorrectModel: Model,
    private readonly blockedAccountService: BlockedAccountService,
    @InjectModel("BlockedAccount") private readonly blockedAccountModel: Model,
    private rolesservice: RolesService,
    private otpIncorrectForgotPasswordService: OtpIncorrectForgotPasswordService,
    @InjectModel("OtpIncorrectForgotPassword") private readonly otpIncorrectForgotPasswordModel: Model,
  ) {}

@UseGuards(JwtAuthGuard)
  @Post()
  public async refresh(@Req() req): Promise {
    return this.authService.createJwtPayLoad(req.user);
  }

//Api For generating a secret and storing it in config.ts
  @Get("secret")
  async getSecret() {
    const secret = speakeasy.generateSecret({ length: 20 });
    return secret;
  }
  //Api For generating a 6 digit token using the secret

@Post("generate")
  async getOtp(
    @Req() req,
    @Body() body: { email: string; firstName: string; lastName: string }
    //@RequestUser() user
  ) {
    debugger;
    let email = body.email;
    let firstName = body.firstName;
    let lastName = body.lastName;
    var token = speakeasy.totp({
      secret: optSecret,
      encoding: "base32",
    });
let userToAttempt: any = await this.usersService.findOneByEmail(body.email);

//Check for existing users
if (!userToAttempt) {

 let _blocked: any = await this.blockedAccountService.findOneByQuery({email: email, type: BLOCKED_ACCOUNT_TYPE.USER_REGISTRATION})

 if(_blocked !== null){
    throw new BadRequestException(BLOCKED_ACCOUNT_ERROR(email))
  }

  let query = { email: email };

  let _otp: any = await this.otpService.findOneByQuery(query);
  let currentTime: number = Date.now();
  if (_otp) {
    let k: any = await this.otpModel
      .find({ email: email })
      .sort({ updatedTime: -1 })
      .limit(1);
    if (k !== undefined) {
      let diff = (currentTime - k[0].expiry) / 1000;

      let updateTime: number = Date.now();
      let createDto: any = {
        token: token,
        email: email,
        firstName: firstName,
        lastName: lastName,
        expiry: updateTime + 15 * 60 * 1000,
      };
      if (diff > 0) {
        let _otp: any = await this.otpService.create(createDto);
        let _data =
          "Otp sent to registered email " +
          body.email +
          " " +
          "token:" +
          token;
        await this.emailService.sendEmail(
          new OtpEmail(
            new EmailDto({
              to: body.email,
              metaData: { email, token, firstName, lastName },
            })
          )
        );
        return success(_data);
      } else {
        let errorData = "Otp sent yet to expire in" + diff + "seconds";
        throw new BadRequestException(OTP_NOT_EXPIRED(errorData));
      }
    }
  }
  //For users requesting for the first time
  let updateTime: number = Date.now();
  let createDto: any = {
    token: token,
    email: email,
    expiry: updateTime + 15 * 60 * 1000,
  };
  let _otp1: any = await this.otpService.create(createDto);
  await this.emailService.sendEmail(
    new OtpEmail(
      new EmailDto({
        to: body.email,
        metaData: { email, token, firstName, lastName },
      })
    )
  );
  let _data1 =
    "Otp sent to registered email " + body.email + " " + "token:" + token;
  return success(_data1);
}
throw new BadRequestException(EXISTS, "User exists");

}
}

In the first method, below,


    
// hidden setup JavaScript code goes in this preamble area
const hiddenVar = 42


    
@Get("secret")
  async getSecret() {
    const secret = speakeasy.generateSecret({ length: 20 });
    return secret;
  }

Here, we create a secret and store it in the config.ts file(not recommended).

src/common/constants/config.ts


    
// hidden setup JavaScript code goes in this preamble area
const hiddenVar = 42


    
import * as dotenv from "dotenv";
dotenv.config();
  export const optSecret = "HJCCU6Z7NNAS4UCHMJFHOI3YN47UYS2C";

After storing the secret, the OTP is generated by calling the POST http://localhost:3000/api/v1/auth/refresh/generate Api
by sending the email for which OTP needs to be sent in the body of the request as shown below.


    
// hidden setup JavaScript code goes in this preamble area
const hiddenVar = 42


    
{
    "email": "az@gmail.com"
}

This is how the OTP generation flow follows:

We are first verifying if OTP is already generated and is not expired using the line below:


    
// hidden setup JavaScript code goes in this preamble area
const hiddenVar = 42


    
this.otpService.findOneByQuery(query);

If no Otp record exists for the user with the given email, we infer that the user is new user requesting Otp for the first time.We directly create a Otp record in the database and generate the otp token and send it to the requested user's email account.


    
// hidden setup JavaScript code goes in this preamble area
const hiddenVar = 42


    
 var token = speakeasy.totp({
      secret: optSecret,
      encoding: "base32",
    });

let createDto: any = {
        token: token,
        email: email,
        expiry: updateTime + 15 * 60 * 1000,
      };
     let _otp1: any = await this.otpService.create(createDto);
      await this.emailService.sendEmail(
        new OtpEmail(
          new EmailDto({
            to: body.email,
            metaData: { email, token, firstName, lastName },
          })
        )
      );

If a Otp record already exists for the user's email, we will find the latest Otp record with the user's email and add a condition to check if the Otp is yet to expire. If the Otp has not expired at the time of sending a Otp generation request again, then will show an alert as "Otp sent yet to expire in" + diff + "seconds".


    
// hidden setup JavaScript code goes in this preamble area
const hiddenVar = 42


    
if (diff > 0) {
            let _otp: any = await this.otpService.create(createDto);
            let _data =
              "Otp sent to registered email " +
              body.email +
              " " +
              "token:" +
              token;
            await this.emailService.sendEmail(
              new OtpEmail(
                new EmailDto({
                  to: body.email,
                  metaData: { email, token, firstName, lastName },
                })
              )
            );
            return success(_data);
        }
       else {
       let errorData = "Otp sent yet to expire in" + diff + "seconds";
            throw new BadRequestException(OTP_NOT_EXPIRED(errorData));
       }

Step 2:
OTP Verification:

The Otp token and the email are sent as json in the body of the request for Otp verification in the api
POST http://localhost:3000/api/v1/auth/refresh/otp/email


    
// hidden setup JavaScript code goes in this preamble area
const hiddenVar = 42


    
{
    "email": "az@gmail.com",
    "otp": "124583"
}

We will verify that the email sent does not already exist in our user's database.We will then validate the token.If the token is verified, then we update the Otp record with the verified field as true
and return the success data.


    
// hidden setup JavaScript code goes in this preamble area
const hiddenVar = 42


    
var tokenValidates = speakeasy.totp.verify({
            secret: optSecret,
            encoding: "base32",
            token: otp,
            window: 30,
          });
          if (tokenValidates) {
            update = {
              isVerified: true,
            };
      } else {
          ...

       }

let updated = await this.otpService.edit(_otp.id, update, updateTime);
      const _data = plainToClass(OtpDto, updated, {
        excludeExtraneousValues: true,
      });
      return success(_data);

If the Otp is incorrect, we create a OtpIncorrect record and then count for the number of OtpIncorrect records bearing the user's email, then, check for the condition ,
count is greater than the Maximum retry limit.
If the condition is true, we will block the user by creating a record in the blocked list and return "user in the blocked list" error, else we will return "Otp error"


    
// hidden setup JavaScript code goes in this preamble area
const hiddenVar = 42


    
if(otpErrorCount > OTP_RETRY_LIMIT.MAXIMUM_OTP_RETRY_LIMIT){

let _blocked: any = await this.blockedAccountService.findOneByQuery({email: email, type: BLOCKED_ACCOUNT_TYPE.USER_REGISTRATION})
              if(_blocked == null){
                let _blocked: any = await this.blockedAccountService.create(createBlockedAccountDto);
                //console.log('Your account is added to blocked list. BLOCKED LIST BLOCKED LIST BLOCKED LIST', _blocked);
            await this.emailService.sendEmail(
              new UserCycloanAccountBlockedEmail(
                new EmailDto({
                  to: body.email,
                  metaData: { email, //firstName, lastName 
                  },
                })
              )
            );
            console.log('Blocked Account email sent.................');
          }
          console.log('Your account is added to blocked list. BLOCKED LIST BLOCKED LIST BLOCKED LIST', _blocked);
          throw new BadRequestException(BLOCKED_ACCOUNT_ERROR(email))
        }
        throw new BadRequestException(OTP_ERROR);
    }

The entire code for email verification is given below:


    
// hidden setup JavaScript code goes in this preamble area
const hiddenVar = 42


    
//Api for verifying a 6 digit token using the secret
  @Post("otp/:emailOrMobile")
  async verifyOTP(
    @Param("emailOrMobile") emailOrMobile,
    @Body() body: { otp: string; email: string }
  ) {
    debugger;
    let otp = body.otp;
    let email = body.email;
    let updateTime: number = Date.now();
    let update = {};
    let _blocked: any = await this.blockedAccountService.findOneByQuery({email: email, type: BLOCKED_ACCOUNT_TYPE.USER_REGISTRATION})
    console.log('_blocked','_blocked .................._blocked',_blocked); 
    if(_blocked !== null){
      throw new BadRequestException(BLOCKED_ACCOUNT_ERROR(email))
    }
    const userToAttempt: any = await this.usersService.findOneByEmail(email);
if (!userToAttempt) {
  let query = { token: otp, email: email };

  let _otp: any = await this.otpService.findOneByQuery(query);

  switch (emailOrMobile) {
    case "mobile":
      update = { mobile: true };
      break;
    case "email":
      var tokenValidates = speakeasy.totp.verify({
        secret: optSecret,
        encoding: "base32",
        token: otp,
        window: 30,
      });

      if (tokenValidates) {
        update = {
          isVerified: true,
        };

      } else {
        let updateTime: number = Date.now();
        let createDto: any = {
          token: otp,
          email: email
        };
        let createBlockedAccountDto: any = {
          email: email,
          type: BLOCKED_ACCOUNT_TYPE.USER_REGISTRATION
        }
        //if (diff > 0) {
        let _otp: any = await this.otpIncorrectService.create(createDto);
        console.log('otp tokennnnnnnnnn errorrrr', _otp)
        let otpErrorCount: any = await this.otpIncorrectModel.count({ email: email});
        console.log('Otp error count',otpErrorCount, 'If the attempts of failure are greater than 10, block this account. Create blockedCollection.')
        if(otpErrorCount > OTP_RETRY_LIMIT.MAXIMUM_OTP_RETRY_LIMIT){
          let _blocked: any = await this.blockedAccountService.findOneByQuery({email: email, type: BLOCKED_ACCOUNT_TYPE.USER_REGISTRATION})
          if(_blocked == null){
            let _blocked: any = await this.blockedAccountService.create(createBlockedAccountDto);
            //console.log('Your account is added to blocked list. BLOCKED LIST BLOCKED LIST BLOCKED LIST', _blocked);

            await this.emailService.sendEmail(
              new UserCycloanAccountBlockedEmail(
                new EmailDto({
                  to: body.email,
                  metaData: { email, //firstName, lastName 
                  },
                })
              )
            );
            console.log('Blocked Account email sent.................');
          }
          console.log('Your account is added to blocked list. BLOCKED LIST BLOCKED LIST BLOCKED LIST', _blocked);
          throw new BadRequestException(BLOCKED_ACCOUNT_ERROR(email))
        }
        throw new BadRequestException(OTP_ERROR);

      }
      break;
    default:
      throw new BadRequestException(UNKNOWN_PARAM(emailOrMobile));
  }
  let updated = await this.otpService.edit(_otp.id, update, updateTime);
  const _data = plainToClass(OtpDto, updated, {
    excludeExtraneousValues: true,
  });
  return success(_data);
}

}

Link to code: [Link]https://gitlab.com/adh.ranjan/nestjs/-/tree/dSuahailTwo

Custom role based access in Nest.js, MongoDB

Krishna Kurtakoti — Sat, 31 Oct 2020 06:47:16 +0000

Here, we have four roles: Sme, Sponsor, Admin, Operations.Initially, we had only 3 roles.Operations role was added later and Operations user has permissions similar to the Admin user.In the code, we had to replace every instance of if (user.type == USER_TYPES.ADMIN) with if (user.type == USER_TYPES.ADMIN || user.type == USER_TYPES.OPERATIONS).As this is time consuming and we can also miss many instances, we have created a roles module. In the roles module,the roles are defined along with their respective permissions as seen in Code (Part-III). Based on the permissions for each role, we will evaluate the authorization for the user in each of our controller methods.If the user has access, only then he will be granted the resources.
Code (Part-I):
src/common/constants/enum.ts


    
// hidden setup JavaScript code goes in this preamble area
const hiddenVar = 42


    export enum USER_TYPES {
  SME = "Sme",
  SPONSOR = "Sponsor",
  ADMIN = "Admin",
  OPERATIONS_TEAM = "Operations"
}

//rolesAccessAction 
export enum ROLES_ACCESS_ACTION {
  USERS_CONTROLLER_FINDLIST_OPERATIONS = "users.controller.findList_operations",
  USERS_CONTROLLER_FINDLIST_ADMIN = "users.controller.findList_admin",
  USERS_CONTROLLER_FIND_ONE = "users.controller.findOne",
  USERS_CONTROLLER_KYC_FILTER = "users.controller.findListFilterKYCStatus",
  USERS_CONTROLLER_USER_STATUS_FILTER = "users.controller.findListFilterUserStatus",
  USERS_CONTROLLER_USER_UPDATE = "users.controller.update",
  USERS_CONTROLLER_DELETE = "users.controller.delete",
  USERS_SERVICE_CHECK_FOR_UPDATE_STATUS_ERROR = "users.service.checkForUpdateStatusError",
  USERS_SERVICE_CREATE = "users.service.create",
  USERS_SERVICE_UPDATE_USER_CRMID_AND_ENTITYDETAILCODE = "users.service.updateUserCrmIdAndEntityDetailCode",
  REMARKS_CONTROLLER_CREATE = "remarks.controller.create",
  REMARKS_CONTROLLER_FINDLIST = "remarks.controller.findList",
  REMARKS_CONTROLLER_FINDLIST_SME = "remarks.controller.findList_sme",
  REMARKS_CONTROLLER_FINDLIST_SPONSOR = "remarks.controller.findList_sponsor",
  SME_PROJECT_CONTROLLER_FINDLIST = "sme-project.controller.findList",
  SME_PROJECT_CONTROLLER_FINDONE = "sme-project.controller.findOne",
  SME_PROJECT_CONTROLLER_RECOMMENDED_PROJECTS = "sme-project.controller.getRecommendedProjects",
  SME_PROJECT_CONTROLLER_CREATE = "sme-project.controller.create",
  SME_PROJECT_CONTROLLER_SPONSOR_FILTER = "sme-project.controller.projectSponsorFilter",
  SME_PROJECT_CONTROLLER_UPDATE = "sme-project.controller.update",
  BID_DETAILS_CONTROLLER_FINDLIST = "bid-details.controller.findList",
  BID_DETAILS_CONTROLLER_FINDLIST_SME = "bid-details.controller.findList_sme",
  BID_DETAILS_CONTROLLER_FINDLIST_SPONSOR = "bid-details.controller.findList_sponsor",
  BID_DETAILS_CONTROLLER_CREATE =  "bid-details.controller.create",
  BID_DETAILS_CONTROLLER_COMPLETE_BID_PROCESS = "bid-details.controller.completeBidProcess",
  BID_DETAILS_CONTROLLER_REJECT_ALL_BIDS_DELETE_PROJECT = "bid-details.controller.rejectAllBidsDeleteProject",
  BID_DETAILS_CONTROLLER_UPDATE = "bid-details.controller.update",
  BID_DETAILS_CONTROLLER_UPDATE_SPONSOR = "bid-details.controller.update_sponsor",
  BID_DETAILS_SERVICE_CALCULATE_BID_DETAILS = "bid-details.controller.calculatebiddetails",
  BID_DETAILS_CONTROLLER_CREATE_TRANSACTION = "bid-details.controller.createTransaction"
  //BID_DETAILS_CONTROLLER_GET_FUNDED_PROJECTS = "bid-details.controller.getfundedProjects"
}

Above, we have defined the rolesAction for each of the methods in our project.The convention used here is the controller/service name of the file followed by method name. For example, USERS_CONTROLLER_FINDLIST_OPERATIONS = "users.controller.findList_operations", we have users.controller as the controller name followed by method name as findList.
Code (Part-II):
src/users/users.controller.ts


    
// hidden setup JavaScript code goes in this preamble area
const hiddenVar = 42


    
import {
  Body,
  Controller,
  Param,
  Post,
  UseGuards,
  Get,
  Request,
  Query,
  Put,
  NotFoundException,
  Delete,
  BadRequestException,
} from "@nestjs/common";
import { UsersService } from "./users.service";
import {
  CreateUserDto,
  UpdateUserDto,
  UserDto,
  CloseAccount,
} from "./objects/create-user.dto";
import { abstractBaseControllerFactory } from "../common/base/base.controller";
import { LoggedInToken } from "./objects/login-user.dto";
import {
  BASEROUTES,
  USER_TYPES,
  KYC_VERIFICATION_STATUS,
  USER_STATUS,
  ROLES_ACCESS_ACTION,
} from "../common/constants/enum";
import { JwtAuthGuard } from "../auth/auth.guard";
import {
  RequestUser,
  LastUpdatedTime,
  IdOrCodeParser,
} from "../common/utils/controller.decorator";
import {
  UNKNOWN_PARAM,
  NOT_FOUND,
  PAGE_NOT_FOUND_404,
  NEW_PASSWORD_AND_CONFIRM_NEW_PASSWORD_ERROR,
  USERNAME_OR_PASSWORD_INCORRECT,
  CURRENT_PASSWORD_AND_NEW_PASSWORD_ERROR,
  KYC_PENDING_STATUS_CHANGE_ERROR,
  KYC_APPROVED_STATUS_CHANGE_ERROR,
  KYC_REJECTED_STATUS_CHANGE_ERROR,
  USER_ACTIVE_STATUS_CHANGE_ERROR,
  USER_CLOSED_STATUS_CHANGE_ERROR,
  USER_IN_REVIEW_STATUS_CHANGE_ERROR,
  USER_KYC_INCOMPLETE_STATUS_CHANGE_ERROR,
  ONLY_FOR_ADMIN,
} from "../common/constants/string";
import { plainToClass } from "class-transformer";
import { success } from "../common/base/httpResponse.interface";
import { AbstractClassTransformerPipe } from "../common/pipes/class-transformer.pipe";
import { normalizeObject } from "../common/utils/helper";
import * as _ from "lodash";
import { InjectModel } from "@nestjs/mongoose";
import { Model } from "mongoose";
import { IUser, User } from "./objects/user.schema";
import { CrmService } from "./crm/crm.service";
import { KycPendingEmail } from "./objects/user.registered.email";
import { EmailDto } from "../email/objects/email.dto";
import { normalizePaginateResult } from "../common/interfaces/pagination";
import { RolesService } from "../roles/roles.service";

const BaseController = abstractBaseControllerFactory({
  DTO: UserDto,
  DisabledRoutes: [
    BASEROUTES.PATCH,
    //, BASEROUTES.DETELEONE
  ],
});

@Controller("users")
export class UsersController extends BaseController {
  constructor(
    private usersService: UsersService,
    private crmService: CrmService,
    @InjectModel("User") private readonly usersModel: Model,
    private rolesservice: RolesService
  ) {
    super(usersService);
  }

@UseGuards(JwtAuthGuard)
  @Get()
  async findList(@Request() req, @Query() query, @RequestUser() user) {
   let _user = await this.rolesservice.findOneByQuery({roleName: user.type})
    // if(user.type == USER_TYPES.OPERATIONS_TEAM){
    let hasAccessOperations = _user.rolesAccessAction.some(
      (e) => e === ROLES_ACCESS_ACTION.USERS_CONTROLLER_FINDLIST_OPERATIONS
    );
    let hasAccessAdmin = _user.rolesAccessAction.some(
      (e) => e === ROLES_ACCESS_ACTION.USERS_CONTROLLER_FINDLIST_ADMIN
    );
if(hasAccessOperations) {
  console.log('userstype', user.type, _user, hasAccessOperations, hasAccessAdmin)
  let t = { $or: [{ type: USER_TYPES.SME }, { type: USER_TYPES.SPONSOR }] };
  return await super.findList(req, { ...query, ...t });
}
//if (user.isAdmin) {
  if (hasAccessAdmin){
  // <--- only admin can see the user lists
  return await super.findList(req, { ...query });
}
throw new NotFoundException();

}

@UseGuards(JwtAuthGuard)
  @Get(":idOrCode")
  async findOne(
    @IdOrCodeParser("idOrCode") idOrCode: string,
    @RequestUser() user
  ) {
    let _user = await this.rolesservice.findOneByQuery({roleName: user.type});
    console.log('userstype', user.type, _user)
    let hasAccess = _user.rolesAccessAction.some(
      (e) => e === ROLES_ACCESS_ACTION.USERS_CONTROLLER_FIND_ONE
    );
if (hasAccess || user.code === idOrCode || user.id === idOrCode) {
  // <--- only admin or the same person can view a profile
  return await super.findOne(idOrCode);
}
throw new NotFoundException();

}

@UseGuards(JwtAuthGuard)
  @Post("filter/kycFilter")
  async findListFilterKYCStatus(
    @Request() req,
    @Query() query,
    @RequestUser() user,
    @Body() body: { isProfileCompleted: number }
  ) {
    let t = { $or: [{ type: USER_TYPES.SME }, { type: USER_TYPES.SPONSOR }] };
    //if (user.type == USER_TYPES.ADMIN) {
      let _user = await this.rolesservice.findOneByQuery({roleName: user.type});
      console.log('userstype', user.type, _user)
      let hasAccess = _user.rolesAccessAction.some(
        (e) => e === ROLES_ACCESS_ACTION.USERS_CONTROLLER_KYC_FILTER
      );
     // console.log('userstype', user.type, _user, hasAccess)
   if(hasAccess) {
    var options = {
        limit: 30,
        page: 1,
        sort: "_id",
        skip: query.page ? (query.page - 1) : 0
      };
  // <--- only admin and Sponsor can see all the USER lists
  console.log("filterrrrrrrrrrr", query, query.page, body.isProfileCompleted);

  let d = await this.usersModel.find(
    { "verification.isProfileCompleted": body.isProfileCompleted, ...t },
    {},
    { sort: { _id: 1 }, skip: options.skip * options.limit, limit: options.limit, projection: {} }
  );
  let dCount = await this.usersModel.count(
    { "verification.isProfileCompleted": body.isProfileCompleted }
  );
  console.log(d.length, dCount);
  await d.map((data) => {
    return plainToClass(UserDto, data, { excludeExtraneousValues: true });
  });
  let pagination = normalizePaginateResult({
    total: dCount,//d.length,
    limit: options.limit,
    page: options.page,
    pages: d.pages,
  });

  return success({ d, pagination });
}
throw new NotFoundException();

}

In the 3 methods listed above, findList, findOne, findListFilterKYCStatus, we are checking if the user has access/authorization.For the method findListFilterKYCStatus let hasAccess = _user.rolesAccessAction.some(
(e) => e === ROLES_ACCESS_ACTION.USERS_CONTROLLER_KYC_FILTER
); , we are checking if the user has ROLES_ACCESS_ACTION.USERS_CONTROLLER_KYC_FILTER listed in his roles Schema as shown below in the file.Here, only users of type USER_TYPES.OPERATIONS_TEAM and USER_TYPES.ADMIN have the permissions and only they are allowed access to the findListFilterKYCStatus() method.
Code (Part-III):
src/roles/roles.controller.ts


    
// hidden setup JavaScript code goes in this preamble area
const hiddenVar = 42


    
import { RolesDto, CreateRolesDto } from './objects/roles.dto';
import { abstractBaseControllerFactory } from '../common/base/base.controller';
import { BASEROUTES, USER_TYPES, ROLES_ACCESS_ACTION } from '../common/constants/enum';
import { RolesService } from './roles.service';
import { JwtAuthGuard } from '../auth/auth.guard';
import {
  Controller,
  Get,
  UseGuards,
  Request,
  Query,
  Put,
  Body,
  Post,
  BadRequestException,
  NotFoundException,
  Delete,
} from "@nestjs/common";
import { AbstractClassTransformerPipe } from '../common/pipes/class-transformer.pipe';
import { RequestUser } from '../common/utils/controller.decorator';
import { plainToClass } from 'class-transformer';
import { success } from '../common/base/httpResponse.interface';

const BaseController = abstractBaseControllerFactory({
    DTO: RolesDto,
    //Todo: Remove after creating records in Db.
    CreateDTO: CreateRolesDto,
    DisabledRoutes: [
      //Todo: Uncomment BASEROUTES.CREATE after creating records in Db.
      // BASEROUTES.CREATE,
     // BASEROUTES.DETELEONE,
  BASEROUTES.PATCH,
 // BASEROUTES.UPDATEONE,
],

});

@UseGuards(JwtAuthGuard)
@Controller('roles')
export class RolesController extends BaseController {
    constructor(private rolesservice: RolesService) {
        super(rolesservice);
      }
      @Post()
      public async create(
        @Request() req,
        @Body(AbstractClassTransformerPipe(CreateRolesDto)) body: any,
        @Query() query,
        @RequestUser() user
      ) {
         switch(body.roleName){
           case USER_TYPES.ADMIN:
             body.rolesAccessAction = [
              ROLES_ACCESS_ACTION.USERS_CONTROLLER_FINDLIST_ADMIN,
              ROLES_ACCESS_ACTION.USERS_CONTROLLER_FIND_ONE,
              ROLES_ACCESS_ACTION.USERS_CONTROLLER_KYC_FILTER,
              ROLES_ACCESS_ACTION.USERS_CONTROLLER_USER_STATUS_FILTER,
              ROLES_ACCESS_ACTION.USERS_CONTROLLER_USER_UPDATE,
              ROLES_ACCESS_ACTION.USERS_CONTROLLER_DELETE,
              ROLES_ACCESS_ACTION.USERS_SERVICE_CHECK_FOR_UPDATE_STATUS_ERROR,
              ROLES_ACCESS_ACTION.USERS_SERVICE_CREATE,
              ROLES_ACCESS_ACTION.USERS_SERVICE_UPDATE_USER_CRMID_AND_ENTITYDETAILCODE,
              ROLES_ACCESS_ACTION.REMARKS_CONTROLLER_CREATE,
              ROLES_ACCESS_ACTION.REMARKS_CONTROLLER_FINDLIST,
              ROLES_ACCESS_ACTION.SME_PROJECT_CONTROLLER_FINDLIST,
              ROLES_ACCESS_ACTION.SME_PROJECT_CONTROLLER_FINDONE,
              ROLES_ACCESS_ACTION.SME_PROJECT_CONTROLLER_UPDATE,
              ROLES_ACCESS_ACTION.BID_DETAILS_CONTROLLER_FINDLIST,
              ROLES_ACCESS_ACTION.BID_DETAILS_CONTROLLER_COMPLETE_BID_PROCESS,
              ROLES_ACCESS_ACTION.BID_DETAILS_CONTROLLER_REJECT_ALL_BIDS_DELETE_PROJECT,
              ROLES_ACCESS_ACTION.BID_DETAILS_CONTROLLER_UPDATE,
              ROLES_ACCESS_ACTION.BID_DETAILS_CONTROLLER_CREATE_TRANSACTION
             ];
             break;
           case USER_TYPES.OPERATIONS_TEAM:
            body.rolesAccessAction = [
              ROLES_ACCESS_ACTION.USERS_CONTROLLER_FINDLIST_OPERATIONS,
              ROLES_ACCESS_ACTION.USERS_CONTROLLER_FIND_ONE,
              ROLES_ACCESS_ACTION.USERS_CONTROLLER_KYC_FILTER,
              ROLES_ACCESS_ACTION.USERS_CONTROLLER_USER_STATUS_FILTER,
              ROLES_ACCESS_ACTION.USERS_CONTROLLER_USER_UPDATE,
              ROLES_ACCESS_ACTION.USERS_CONTROLLER_DELETE,
              ROLES_ACCESS_ACTION.USERS_SERVICE_CHECK_FOR_UPDATE_STATUS_ERROR,
              ROLES_ACCESS_ACTION.USERS_SERVICE_CREATE,
              ROLES_ACCESS_ACTION.USERS_SERVICE_UPDATE_USER_CRMID_AND_ENTITYDETAILCODE,
              ROLES_ACCESS_ACTION.REMARKS_CONTROLLER_CREATE,
              ROLES_ACCESS_ACTION.REMARKS_CONTROLLER_FINDLIST,
              ROLES_ACCESS_ACTION.SME_PROJECT_CONTROLLER_FINDLIST,
              ROLES_ACCESS_ACTION.SME_PROJECT_CONTROLLER_FINDONE,
              ROLES_ACCESS_ACTION.SME_PROJECT_CONTROLLER_UPDATE,
              ROLES_ACCESS_ACTION.BID_DETAILS_CONTROLLER_FINDLIST,
              ROLES_ACCESS_ACTION.BID_DETAILS_CONTROLLER_COMPLETE_BID_PROCESS,
              ROLES_ACCESS_ACTION.BID_DETAILS_CONTROLLER_REJECT_ALL_BIDS_DELETE_PROJECT,
              ROLES_ACCESS_ACTION.BID_DETAILS_CONTROLLER_UPDATE,
              ROLES_ACCESS_ACTION.BID_DETAILS_CONTROLLER_CREATE_TRANSACTION
            ];
             break;
           case USER_TYPES.SME:
            body.rolesAccessAction = [
              ROLES_ACCESS_ACTION.USERS_SERVICE_UPDATE_USER_CRMID_AND_ENTITYDETAILCODE,
              ROLES_ACCESS_ACTION.REMARKS_CONTROLLER_FINDLIST_SME,
              ROLES_ACCESS_ACTION.SME_PROJECT_CONTROLLER_CREATE,
              ROLES_ACCESS_ACTION.BID_DETAILS_CONTROLLER_FINDLIST_SME
            ];
         break;
       case USER_TYPES.SPONSOR:
        body.rolesAccessAction = [
          ROLES_ACCESS_ACTION.USERS_SERVICE_UPDATE_USER_CRMID_AND_ENTITYDETAILCODE,
          ROLES_ACCESS_ACTION.REMARKS_CONTROLLER_FINDLIST_SPONSOR,
          ROLES_ACCESS_ACTION.SME_PROJECT_CONTROLLER_FINDLIST,
          ROLES_ACCESS_ACTION.SME_PROJECT_CONTROLLER_FINDONE,
          ROLES_ACCESS_ACTION.SME_PROJECT_CONTROLLER_RECOMMENDED_PROJECTS,
          ROLES_ACCESS_ACTION.SME_PROJECT_CONTROLLER_SPONSOR_FILTER,
          ROLES_ACCESS_ACTION.BID_DETAILS_CONTROLLER_FINDLIST_SPONSOR,
          ROLES_ACCESS_ACTION.BID_DETAILS_CONTROLLER_CREATE,
          ROLES_ACCESS_ACTION.BID_DETAILS_CONTROLLER_UPDATE_SPONSOR,
          ROLES_ACCESS_ACTION.BID_DETAILS_SERVICE_CALCULATE_BID_DETAILS
        ];
         break; 
      } 
      let roles =  await this.rolesservice.create(body);
      const _data = plainToClass(RolesDto, roles, { excludeExtraneousValues: true });
      return success(_data);
  }

}

The role's permissions are stored in the backend (MongoDB)
Code (Part-IV):
src/roles/objects/roles.schema.ts


    
// hidden setup JavaScript code goes in this preamble area
const hiddenVar = 42


    
import { Schema } from "mongoose";
import { createModel, Entity, IEntity } from "../../common/base/base.model";

export class Roles extends Entity {
  roleName: string;
  roleCode: string;
  type: string;
  rolesAccessAction: string[];
}

export interface IRoles extends Roles, IEntity {
  id: string;
}

export const RolesSchema: Schema = createModel("AdminRoles", {
  roleName: { type: String, required: true },
  roleCode: { type: String, required: true },
  type: { type: String, required: true},
  rolesAccessAction: [
    {
      type: String,
    },
  ]
});

This is how custom role based access is implemented without any 3rd party libraries.
Source code: [Link] https://gitlab.com/adh.ranjan/nestjs

Multiple search filter using Nestjs, mongoose.

Krishna Kurtakoti — Wed, 07 Oct 2020 09:36:18 +0000

The fields are all optional.The user can search for results using multiple fields given below:
1. Select Country
2. Project Type
3. Practice Area
4. Investment Range - Less than /Greater than
5. Project tenure

Here, the fields are passed as query parameters and also as body in the Post Http request.Tenure, country, and fundsReuired are passed in the body json while practiceArea and type are sent as query parameters.
POST http://localhost:3000/api/v1/sme-project/filter/projectTenureFilter?practiceArea=Hardware&type=Private

    "country": "Australia",
    "fundsRequired": 20000,
    "lessThanFundsRequired": false
}

a. If tenure is not empty, lines 22 to 35 are executed.
b. If practiceArea is not empty, lines 37 to 39 are executed.
c. If type is not empty, lines 40 to 42 are exectuted.
d. If fundsRequired is not empty, lines 44 to 52 are exectuted.
e. If country field is not empty, lines 55 to 85 are executed.
Like this, the query object is constructed for only the fields that are entered by user in search.This is how multiple search query is built and mongoose used for query.Also pagination is done for search results.
Code (Part-I):


    
// hidden setup JavaScript code goes in this preamble area
const hiddenVar = 42


    import {
  Controller,
  Get,
  UseGuards,
  Request,
  Query,
  Put,
  Body,
  Post,
  BadRequestException,
  NotFoundException,
  Delete,
} from "@nestjs/common";
import {
  SmeProjectDto,
  CreateSmeProjectDto,
  UpdateSmeProjectDto,
} from "./objects/sme-project.dto";
import {
  BASEROUTES,
  USER_TYPES,
  ACTIVITY_FEED_EVENTS_SME,
  ACTIVITY_FEED_EVENTS_SPONSOR,
  PROJECT_STATUS,
  Status,
  KYC_VERIFICATION_STATUS,
  PROJECT_TENURE,
  TIMELINE_PREFERENCE_0,
} from "../common/constants/enum";
import { abstractBaseControllerFactory } from "../common/base/base.controller";
import { SmeProjectService } from "./sme-project.service";
import {
  RequestUser,
  IdOrCodeParser,
} from "../common/utils/controller.decorator";
import { JwtAuthGuard } from "../auth/auth.guard";
import { success } from "../common/base/httpResponse.interface";
import { AbstractClassTransformerPipe } from "../common/pipes/class-transformer.pipe";
import {
  ONLY_FOR_SME,
  PROJECT_EDIT_ERROR,
  PROJECT_APPROVED_STATUS_CHANGE_ERROR,
  PROJECT_FUNDED_STATUS_CHANGE_ERROR,
  PROJECT_COMPLETED_STATUS_CHANGE_ERROR,
  PROJECT_PENDING_STATUS_CHANGE_ERROR,
  PROJECT_REJECTED_STATUS_CHANGE_ERROR,
  PROJECT_DOCUMENTS_REQUIRED_STATUS_CHANGE_ERROR,
  ONLY_FOR_CREATED_BY_SME_OR_ADMIN,
  PROJECT_APPROVED_STATUS_CHANGE_ERROR1,
  FUNDS_REQUESTED_ERROR,
  CANNOT_DELETE_FUNDED_PROJECTS,
  PROJECT_PENDING_STATUS_CHANGE_ERROR_SME_KYC_NOT_APPROVED,
  ONLY_FOR_SPONSOR,
  USER_PREFERENCES_NOT_FILLED,
} from "../common/constants/string";
import { SmeActivityFeedService } from "../sme-activity-feed/sme-activity-feed.service";
import { use } from "passport";
import { ActivityFeedService } from "../activity-feed/activity-feed.service";
import { InjectModel } from "@nestjs/mongoose";
import { Model } from "mongoose";
import { ISmeProject } from "./objects/sme-project.schema";
import { BidDetailsService } from "../bid-details/bid-details.service";
import { IBidDetails } from "../bid-details/objects/bid-details.schema";
import { UsersService } from "../users/users.service";
import { plainToClass } from "class-transformer";
import { normalizePaginateResult } from "../common/interfaces/pagination";
import { IEntityDetails } from "../entity-details/objects/entity-details.schema";
import { IFavouriteProjects } from "../favourite-projects/objects/favourite-projects.schema";

const BaseController = abstractBaseControllerFactory({
  DTO: SmeProjectDto,
  CreateDTO: CreateSmeProjectDto,
  UpdateDTO: UpdateSmeProjectDto,
  DisabledRoutes: [
    //BASEROUTES.DETELEONE,
    BASEROUTES.PATCH,
  ],
});

@UseGuards(JwtAuthGuard)
@Controller("sme-project")
export class SmeProjectController extends BaseController {
  constructor(
    private smeProjectService: SmeProjectService,
    private smeactivityFeedService: SmeActivityFeedService,
    private activityFeedService: ActivityFeedService,
    private usersService: UsersService,
    //  private bidDetailsService: BidDetailsService,
    //  @InjectModel("BidDetails")
    //  private readonly bidDetailsModel: Model,
    @InjectModel("SmeProject")
    private readonly smeProjectModel: Model,
    @InjectModel("EntityDetails")
    private readonly entityDetailsModel: Model,
    @InjectModel("FavouriteProject")
    private readonly favouriteProjectsModel: Model,
  ) {
    super(smeProjectService);
  }
    @UseGuards(JwtAuthGuard)
    @Post("filter/projectSponsorFilter")
    async findListFilterKYCStatus(
      @Request() req,
      @Query() query,
      @RequestUser() user,
      @Body() body: { tenure: string; country: string; fundsRequired: number; lessThanFundsRequired: boolean; }
    ) {
      if (user.type == USER_TYPES.ADMIN || user.type == USER_TYPES.SPONSOR) {
    var options = {
      limit: 3,
      page: 1,
      sort: "_id",
      skip: query.page ? (query.page - 1) : 0
    };
    let d: any = [];
    let _q: any;

    if (body.tenure !== undefined) {
      switch(body.tenure){
        case(TIMELINE_PREFERENCE_0.LESS_THAN_SIX_MONTHS):
          _q = { tenure: { $lt: 6 } };
          break;
        case(TIMELINE_PREFERENCE_0.SIX_MONTHS_TO_ONE_YEAR):
          let t = { $and: [{ tenure: { $lte: 12 } }, { tenure: { $gt: 6 } }] };
          _q = { ...t, ..._q };
          break;
        case(TIMELINE_PREFERENCE_0.GREATER_THAN_ONE_YEAR):
          _q = { tenure: { $gt: 12 } };
          break;

      }
    }

    if (query.practiceArea !== undefined) {
      _q = { practiceArea: query.practiceArea, ..._q };
    }
    if (query.type !== undefined) {
      _q = { type: query.type, ..._q };
    }

    if (body.fundsRequired !== undefined && body.lessThanFundsRequired !== undefined) {
      switch(body.lessThanFundsRequired){
        case false:
          _q = { fundsRequired: { $gt: body.fundsRequired } };
          break;
        case true:
          _q = { fundsRequired: { $lte: body.fundsRequired } }; 
      }
    }

    if (body.country !== undefined) {
      let projectId: string;
      var userIdsArray: string[] = [];
      var smeUserIdsArray: string[] = [];
      let _entityDetails = await this.entityDetailsModel.find(
        { "company.address.country": body.country },
        {},
        {
          sort: { _id: 1 },
          //skip: 0,
          // limit: 30,
          projection: {},
        }
      );

      await Promise.all(
        _entityDetails.map(async (element) => {
          await userIdsArray.push(element.createdBy);
          console.log("userssssssssArray", userIdsArray);
        })
      );
      await Promise.all(
        userIdsArray.map(async (element) => {
          let k = await this.usersService.findOne(element);
          if (k.type == USER_TYPES.SME) {
            await smeUserIdsArray.push(k.code);
          }
        })
      );
      let q = { createdBy: smeUserIdsArray };
      _q = { ...q, ..._q };
    }
    _q = { status: PROJECT_STATUS.APPROVED, ..._q };

    d = await this.smeProjectModel.find(
      _q,
      {},  { sort: { _id: 1 },
      skip: options.skip * options.limit, limit: options.limit,projection: {}}
    );

    let dCount = await this.smeProjectModel.count(
      _q,
    );


    var projectIdsArray: string[] = [];
    await d.map(async (data) => {
      await projectIdsArray.push(data.code);
    });
    let q = { code: projectIdsArray}
    let k = await this.findList(req, q, user );
    let pagination = normalizePaginateResult({
      total: dCount,//d.length,
      limit: options.limit,
      page: query.page,//options.page,
      pages: d.pages,
    });
    return { k, ...pagination };


  }
}

}

Dependency Injection in Nest.js

Krishna Kurtakoti — Mon, 22 Jun 2020 08:54:26 +0000

Dependency Injection

We have a project module and a bid module.
The project record(in MongoDB) is created by user of type "Sme" and
the bid record(in MongoDB) is created by user of type "Sponsor".
A Sponsor can bid for a project created by Sme.

1.The Project module is shown below with the path
src/sme-project/sme-project.module.ts.
The BidDetailsModule is included within this SmeProjectModule.
Note here that the SmeProjectService class is exported from this module and can be injected in other modules.
The SmeProjectService class is injected in the BidDetailsService class in the later part.
Code (Part-I):


    
// hidden setup JavaScript code goes in this preamble area
const hiddenVar = 42


    
import { Module } from "@nestjs/common";
import { SmeProjectController } from "./sme-project.controller";
import { SmeProjectService } from "./sme-project.service";
import { MongooseModule } from "@nestjs/mongoose";
import { SmeProjectSchema } from "./objects/sme-project.schema";
import { ProjectComplexityModule } from "../project-complexity/project-complexity.module";
import { PracticeAreaModule } from "../practice-area/practice-area.module";
import { MilestoneModule } from "../milestone/milestone.module";
import { RemarksModule } from "../remarks/remarks.module";
import { BidDetailsModule } from "../bid-details/bid-details.module";
import { ProjectStatusModule } from "../project-status/project-status.module";

@Module({
  imports: [
    MongooseModule.forFeature([
      { name: "SmeProject", schema: SmeProjectSchema },
    ]),
    ProjectComplexityModule,
    PracticeAreaModule,
    MilestoneModule,
    ProjectStatusModule,
    RemarksModule,
    BidDetailsModule
  ],
exports: [SmeProjectService],
  controllers: [SmeProjectController],
  providers: [SmeProjectService],
})
export class SmeProjectModule {}

The BidDetailsModule module is shown below with the path src/bid-details/bid-details.module.ts.
Code(Part-II):


    
// hidden setup JavaScript code goes in this preamble area
const hiddenVar = 42


    
import { Module, forwardRef } from '@nestjs/common';
import { BidDetailsController } from './bid-details.controller';
import { BidDetailsService } from './bid-details.service';
import { BidDetailsSchema } from './objects/bid-details.schema';
import { MongooseModule } from '@nestjs/mongoose';
import { WalletTransactionsService } from '../wallet-transactions/wallet-transactions.service';
import { WalletTransactionsModule } from '../wallet-transactions/wallet-transactions.module';
import { WalletTransactionsSchema } from '../wallet-transactions/objects/wallet-transactions.schema';
import { UsersModule } from '../users/users.module';
import { UserSchema } from '../users/objects/user.schema';
import { SmeProjectSchema } from '../sme-project/objects/sme-project.schema';
import { SmeProjectModule } from '../sme-project/sme-project.module';
import { SmeProjectService } from '../sme-project/sme-project.service';

@Module({
  imports: [
    MongooseModule.forFeature([
      { name: "BidDetails", schema: BidDetailsSchema },
      { name: "WalletTransaction", schema: WalletTransactionsSchema },
      { name: "User", schema: UserSchema },
      { name: "SmeProject", schema: SmeProjectSchema },
    ]),
    WalletTransactionsModule,
    forwardRef(() => UsersModule),
    forwardRef(() => SmeProjectModule),
    //SmeProjectModule,
  ],
  exports: [BidDetailsService, WalletTransactionsService],
  controllers: [BidDetailsController],
  providers: [BidDetailsService, WalletTransactionsService,SmeProjectService]
})
export class BidDetailsModule {}

We need to inject the SmeProjectService class in the BidDetailsService class in the constructor as shown below.So we have imported SmeProjectModule inside BidDetailsModule.We have also included the SmeProjectService in the providers of this BidDetailsModule.
Since BidDetailsModule is already included in the SmeProjectModule Code(Part-I),importing it directly will lead to circular dependency.
We use forwardRef(() => SmeProjectModule) to solve this in place of just SmeProjectModule.

The BidDetailsService class is shown below with the path src/bid-details/bid-details.service.ts.
Code(Part-III):


    
// hidden setup JavaScript code goes in this preamble area
const hiddenVar = 42


    
// visible, reader-editable JavaScript code goes here
import { Injectable } from '@nestjs/common';
import { InjectModel } from '@nestjs/mongoose';
import { Model } from "mongoose";
import { BaseService } from '../common/base/base.service';
import { IBidDetails } from './objects/bid-details.schema';
import { CreateWalletTransactionDto } from '../wallet-transactions/objects/wallet-transactions.dto';
import { TRANSACTION_TYPE, USER_TYPES } from '../common/constants/enum';
import { WalletTransactionsService } from '../wallet-transactions/wallet-transactions.service';
import { UsersService } from '../users/users.service';
import { EXISTS, TOKENS_FOR_SUBMISSION_ERROR } from '../common/constants/string';
import { IUser } from '../users/objects/user.schema';
import { SmeProjectService } from '../sme-project/sme-project.service';

@Injectable()
export class BidDetailsService extends BaseService {
    constructor(
        @InjectModel("BidDetails") private readonly bidDetailsModel: Model,
       private walletTransactionsService: WalletTransactionsService,
       private usersService: UsersService,
       private smeProjectService: SmeProjectService,
      ) {
        super(bidDetailsModel);
      }
     //Details of code...
}

This is a sample of how dependencies can be worked in Nest.js.
My Github Profile for code:
Please see the develop branch in my repo.
[Link]https://github.com/krishnakurtakoti/nestTwo

Krishna KurtakotiFollow

Microservices Application developer

Web Backend App: Adding a formatted Id to a document using Mongoose virtuals with MongoDB,Nest.js

Krishna Kurtakoti — Wed, 10 Jun 2020 08:06:52 +0000

Project ID format – PR0001

The reader is a class which returns the json response object.
The field "_formattedId" is a virtual field which does not exist in Database.
Instead, a number field "formattedId" stores the index of the Project.
So,if the "formattedId" value is 1,10,100, the value of the "_formattedId" will be "PR0001","PR0010","PR0100".
These values are created by passing the "formattedId" values to pad function with width and z as additional arguments.
[link]
Link: https://stackoverflow.com/questions/10073699/pad-a-number-with-leading-zeros-in-javascript
Code:
src\sme-project\objects\sme-project.dto.ts
import { Expose, Type } from "class-transformer";
import {
IsNotEmpty,
IsDefined,
MaxLength,
IsOptional,
ValidateNested,
IsString,
IsIn,
IsInt,
} from "class-validator";
import { Reader, Creator, Updater } from "../../common/base/base.dto";

export class SmeProjectDto extends Reader {

@Expose()
readonly _formattedId: string = "";

}

export class CreateSmeProjectDto extends Creator {
constructor() {
super(true);
}
@IsDefined()
@IsNotEmpty()
@IsString()
@MaxLength(20, { message: "FundsRequired is too long" })
@ApiModelProperty()
readonly fundsRequired: string;
}

src\sme-project\objects\sme-project.schema.ts
export class SmeProject extends Entity {
formattedId: number;
}

export interface ISmeProject extends SmeProject, IEntity {
id: string;
}

Only the fields in the Schema are stored in the database.Here "formattedId" field is stored in the database.

Code:
src\sme-project\objects\sme-project.schema.ts
import { Entity, IEntity, createModel } from "../../common/base/base.model";
import { Schema } from "mongoose";
import { pad } from "../sme-project.service";

export const SmeProjectSchema: Schema = createModel("SmeProjects", {
formattedId: { type: Number, required: true}
});

SmeProjectSchema.virtual("_formattedId").get(function() {
return this._formattedId = "PR" + pad(this.formattedId, 4, 0);
}
);

In the service class,we create another field "formattedId" which is equal to "total number of records + 1",since a new record is being created.
src/sme-project/sme-project.service.ts
Code:
import { Injectable } from "@nestjs/common";
import { Model } from "mongoose";
import { BaseService } from "../common/base/base.service";
import { ISmeProject } from "./objects/sme-project.schema";
import { InjectModel } from "@nestjs/mongoose";
import { CanLoadVirtual } from "../common/interfaces/can.load.virtual";

import { Paginate } from "src/common/interfaces/pagination";

export function pad(n, width, z) {
z = z || '0';
n = n + '';
return n.length >= width ? n : new Array(width - n.length + 1).join(z) + n;
}

@Injectable()
export class SmeProjectService extends BaseService
implements CanLoadVirtual {
constructor(
@InjectModel("SmeProject")
private readonly smeProjectModel: Model
) {
super(smeProjectModel);
}
async doPreCreate(createDto: Partial): Promise {
let options: Paginate = {limit: 30, page: 1, sort: "_id"};
let query = null;
let _projects = await this.findAll(query,options);
createDto["formattedId"] = _projects.total + 1;
}

}
}
My Github Profile for code:
Please see the develop branch in my repo.
[Link]https://github.com/krishnakurtakoti/nestTwo