In this post, we will see how to use different visualizations, like the simple graph, pie chart, world map panel in the grafana dashboard by writing queries in Influx query language and fetching data from the InfluxDB database which is a time-series database.
a. Data Visualization
b. Installing Plugins Docker Grafana
c. Writing to the InfluxDB2.0
LINE CHARTS:
1.
As we can see, we are showing the records for 2 different blocks, i.e, DS_Id = 0 and DS_Id = 1.The timestamp is the same for both the blocks, 1598918400, 1598918402,
1598918403 ….. .The date for the above can be obtained from the link: https://www.epochconverter.com/
| TIMESTAMP | DS_Id | POWER_A | POWER_B | POWER_C |
|---|---|---|---|---|
| 1598918400 | 0 | 1403.421 | 712.372 | 1680.471 |
| 1598918402 | 0 | 1423.817 | 731.249 | 1680.658 |
| 1598918403 | 0 | 1444.172 | 749.339 | 1700.859 |
| 1598918404 | 0 | 1774.402 | 1106.427 | 2041.954 |
| 1598918405 | 0 | 1774.402 | 1106.427 | 2041.954 |
| TIMESTAMP | DS_Id | POWER_A | POWER_B | POWER_C |
|---|---|---|---|---|
| 1598918400 | 1 | 821.847 | 574.748 | 1203.807 |
| 1598918402 | 1 | 823.367 | 574.315 | 1203.795 |
| 1598918403 | 1 | 819.939 | 574.261 | 1203.647 |
| 1598918404 | 1 | 819.939 | 574.261 | 1203.647 |
Our requirement is to get the aggregated power for POWER_A, POWER_B, POWER_C fields.
For example, for the timestamp 1598918400,
| TIMESTAMP | DS_Id | POWER_A | POWER_B | POWER_C |
|---|---|---|---|---|
| 1598918400 | 0 | 1403.421 | 712.372 | 1680.471 |
| 1598918400 | 1 | 821.847 | 574.748 | 1203.807 |
we have values for POWER_A as 1403.421W and 821.847W, sum as 2225.268W. Likewise, we have to calculate for all the time-series values(1598918402, 1598918403, 1598918404, …) for POWER_A, POWER_B and POWER_C also.
This computation has to be done by Grafana. We are implementing this using the SQL syntax like queries as shown below:
We are going to compute the aggregated power for the field POWER_A now.
All the queries were constructed and executed successfully by Narendra Reddy Mallidi, SQL Developer.
We have our first query below:
SELECT POWER_A as AA FROM "hdb2" WHERE "DS_ID" = '0'
Here, POWER_A is the variable where are going to fetch from the table(table is called measurement in InfluxDB queries) named "hdb2".
The same thing is followed for other 4 blocks also(DS_ID =’1’, DS_ID=’2’, DS_ID=’3’, DS_ID=’4’)
SELECT POWER_A as BB FROM "hdb2" WHERE "DS_ID" = '1'
SELECT POWER_A as CC FROM "hdb2" WHERE "DS_ID" = '2'
SELECT POWER_A as DD FROM "hdb2" WHERE "DS_ID" = '3'
SELECT POWER_A as EE FROM "hdb2" WHERE "DS_ID" = '4'
Now, we compute the aggregated power for POWER_A with the below query:
SELECT sum(ds1) + sum(ds0) + sum(ds2) + sum(ds3) + sum(ds4) AS Total
FROM
(
SELECT POWER_A as ds0 FROM "hdb2" WHERE "DS_ID" = '0'
),
(
SELECT POWER_A as ds1 FROM "hdb2" WHERE "DS_ID" = '1'
),
(
SELECT POWER_A as ds2 FROM "hdb2" WHERE "DS_ID" = '2'
),
(
SELECT POWER_A as ds3 FROM "hdb2" WHERE "DS_ID" = '3'
),
(
SELECT POWER_A as ds4 FROM "hdb2" WHERE "DS_ID" = '4'
) WHERE $timeFilter GROUP BY time(1s) fill(null)
Here, hdb2 is the table name in our INFLUXDB1.8 database from where we are fetching our data into the Grafana dashboard running at port 3000.
We get the graph as shown below:
| TIMESTAMP | SAMPLE | POWER_A |
|---|---|---|
| 2020-09-01 08:27:16 | Sample 1 | 7.92k |
| TIMESTAMP | SAMPLE | POWER_A |
|---|---|---|
| 2020-09-01 08:17:08 | Sample 2 | 5.77k |
(Sample 2):
TIMESTAMP: 2020-09-01 08:17:08
Aggregate_Power: 5.77k
We are getting the correct aggregated values for POWER_A. We need to do the same for POWER_B and POWER_C.
For POWER_B:
SELECT sum(ds1) + sum(ds0) + sum(ds2) + sum(ds3) + sum(ds4) AS PB
FROM
(
SELECT POWER_B as ds1 FROM "hdb2" WHERE "DS_ID" = '1'
),
(
SELECT POWER_B as ds0 FROM "hdb2" WHERE "DS_ID" = '0'
),
(
SELECT POWER_B as ds2 FROM "hdb2" WHERE "DS_ID" = '2'
),
(
SELECT POWER_B as ds3 FROM "hdb2" WHERE "DS_ID" = '3'
),
(
SELECT POWER_B as ds4 FROM "hdb2" WHERE "DS_ID" = '4'
) WHERE $timeFilter GROUP BY time(1s) fill(null)
For POWER_C:
SELECT sum(ds1) + sum(ds0) + sum(ds2) + sum(ds3) + sum(ds4) AS PC
FROM
(
SELECT POWER_C as ds1 FROM "hdb2" WHERE "DS_ID" = '1'
),
(
SELECT POWER_C as ds0 FROM "hdb2" WHERE "DS_ID" = '0'
),
(
SELECT POWER_C as ds2 FROM "hdb2" WHERE "DS_ID" = '2'
),
(
SELECT POWER_C as ds3 FROM "hdb2" WHERE "DS_ID" = '3'
),
(
SELECT POWER_C as ds4 FROM "hdb2" WHERE "DS_ID" = '4'
) WHERE $timeFilter GROUP BY time(1s) fill(null)
We will get the graph as shown below:
| TIMESTAMP | SAMPLE | POWER_A |
|---|---|---|
| 2020-09-01 08:27:12 | Sample 3 | 7.89k |
(Sample 3):
TIMESTAMP: 2020-09-01 08:27:12
Aggregate_Power: 7.89k
| TIMESTAMP | SAMPLE | POWER_A |
|---|---|---|
| 2020-09-01 08:17:08 | Sample 4 | 5.77k |
(Sample 4):
TIMESTAMP: 2020-09-01 08:17:08
Aggregate_Power: 5.77k
As we can see, these aggregated values for POWER_A(sample 3 and 4) confirm with the values shown in the previous graphs (sample 1 and 2) with the same timestamp.
PIE CHART VISUALIZATION:
Below is the query for visualization of 3 components:
Lift_Energy, Public_Lighting_Energy, Booster_Pump_Energy.
**
SELECT
(( (sum("LIFT_TOTAL")) / (($to - $from) / 1000) ) )* (($to - $from) / 3600000)
as Lift_Energy,
(((sum("LIGHT_TOTAL")) / (($to - $from) / 1000))) * ((($to - $from) / 1000) / 3600)
as Public_Lighting_Energy,
(((sum("PUMP_TOTAL")) / (($to - $from) / 1000))) * ((($to - $from) / 1000) / 3600) as Booster_Pump_Energy
FROM "hdb2" WHERE $timeFilter GROUP BY time(24h)
**
Here, we have combined 3 subqueries.
- (sum("LIFT_TOTAL")) / (($to - $from) / 1000) ) )* (($to - $from) / 3600000) as Lift_Energy
- (sum("LIGHT_TOTAL")) / (($to - $from) / 1000))) * ((($to - $from) / 1000) / 3600) as Public_Lighting_Energy
- (sum("PUMP_TOTAL")) / (($to - $from) / 1000))) * ((($to - $from) / 1000) / 3600) as Booster_Pump_Energy All the 3 queries are similar. We will take 1st query. The records are taken for each second,i.e, for every consecutive second we are recording the data. Here, (sum("LIFT_TOTAL")) is the sum computed over the period mentioned - (($to - $from) in the time window.
The query has been updated by Grafana as:
SELECT
(( (sum("LIFT_TOTAL")) / ((1599190800000 - 1598898600000) / 1000) ) )* ((1599190800000 - 1598898600000) / 3600000)
as Lift_Energy,
(((sum("LIGHT_TOTAL")) / ((1599190800000 - 1598898600000) / 1000))) * (((1599190800000 - 1598898600000) / 1000) / 3600)
as Public_Lighting_Energy,
(((sum("PUMP_TOTAL")) / ((1599190800000 - 1598898600000) / 1000))) * (((1599190800000 - 1598898600000) / 1000) / 3600) as Booster_Pump_Energy
FROM "hdb2" WHERE time >= 1598898600000ms and time <= 1599190800000ms GROUP BY time(24h)
Since the precision is in milliseconds, we are dividing it by 1000.Now, we get total power for the time range applied. The unit for energy consumption is watt-hour(Wh). Example, a 40-watt electric appliance operating continuously for 25 hours uses one kilowatt-hour. The value ((($to - $from) / 1000) / 3600) gives the total operating hours which multiplied with the total power,i.e,(sum("PUMP_TOTAL")) / (($to - $from) / 1000))) gives power consumption in watt-hour units.
WORLD MAP VISUALIZATION:
The sample csv table we are using for the world map is shown below.
| Time | DS_ID | graphStatement | latitude | longitude | value | totalPower |
| ------------|:-----:| -----------------------------------: | -------: | --------: |
----- |:----------:|
| 2020-12-27 16:02:27 | 4 | Total: 49836, lift: 18377, light: 3357, pump: 54699339, DS_ID: 4 | 1.3 | 104 | 1 | 49836 |
Above is the csv file which we are uploading in the influxdb into the "hdb7" measurement.
Query for world map:
SELECT * from "hdb7" GROUP BY * ORDER BY DESC LIMIT 4
We can see the query and the settings also in the image below. Here, we are using latitude and longitude values to plot the map and put the graphStatement field in the label of the map.
b. More on Installing Plugins Docker Grafana:
- Open the SSH terminal on your machine and run the following command:
ssh your_username@host_ip_address
If the username on your local machine matches the one on the server you are trying to connect to, you can just type:
ssh host_ip_address
And hit Enter.
After successful login, execute the below commands in the shell:
sudo docker ps -a
sudo docker exec -it --user=root grafana /bin/sh
grafana-cli plugins install grafana-worldmap-panel
sudo docker container stop d1ead747ec87
sudo docker start d1ead747ec87
1.
The sudo docker ps -a lists all the running containers.
2.
We can execute/test the commands for the application running inside the container with sudo docker exec -it --user=root grafana /bin/sh command.
We can also ping to test the port connections with the commands:
curl http://localhost:3000
curl http://160.100.100.204:3000 where the ip-address is of the remote virtual machine provisioned in the cloud which we have logged into.
3.
The plugins are installed with grafana-cli plugins install grafana-worldmap-panel.
4.
The docker conatiners are restarted with commands:
sudo docker container stop d1ead747ec87
sudo docker start d1ead747ec87
c. Writing to the InfluxDB2.0
Reference:
Requirements:
-
Get started with InfluxDB 2.0
The InfluxDB 2.0 time series platform is purpose-built to collect, store, process and visualize metrics and events. Get started with InfluxDB OSS v2.0 by downloading InfluxDB, installing the necessary executables, and running the initial setup process.
If not installed, follow the link
https://docs.influxdata.com/influxdb/v2.0/get-started/Start InfluxDB by running the influxd daemon:
k@k-Lenovo-G50-70:~$ influxd
-
Python installed
k@k-Lenovo-G50-70:~$ python --version
Python 2.7.12
k@k-Lenovo-G50-70:~$ python3 --version
Python 3.5.2
Steps:
-
Replace the values of INFLUX_TOKEN, ORG, BUCKET_NAME and measurement_name with the name of the table you need to create.
Also replace the csv path you need to upload the csv file at line:
with open('/home/k/Downloads/influxData/data_0_20200901.csv') as csv_file:
In the csv file, we have time stored in Unix Timestamp format. Run the below program:
k@k-Lenovo-G50-70:~/a_b$ python3 pushToInflux2_0.py
/home/k/a_b/pushToInflux2_0.py
Code:
- In the InfluxDB2.0 screen page at: http://localhost:8086/orgs/772821f18f2f7a0d/data-explorer?bucket=b Under the Explore option
We need to set the WINDOW PERIOD as 15s / 1m to see more points on the dashboard.
Also, the timing window must be adjusted according to the timestamp of the record.
For e.g,
| TIMESTAMP | DS_Id | POWER_A | POWER_B | POWER_C |
|---|---|---|---|---|
| 1598918400 | 0 | 1403.421 | 712.372 | 1680.471 |
So, we need to apply the time range from the above date for the results to show as in the above window.
My Github Profile for code:
Please see the master branch in my repo.
[Link]https://github.com/krishnakurtakoti/python-influxdb-2.0-write
Top comments (0)