DEV Community: Alan

Sensors Dashboard (MQTT Raspberry PI+ESP32+FastHTML)

Alan — Fri, 07 Feb 2025 07:07:17 +0000

This guide explains how to set up an ESP32 microcontroller with an LM35 temperature sensor to send real-time data via MQTT to a Raspberry Pi running a Mosquitto broker. The data is then visualized using a sensor dashboard with an interactive UI.

We'll walk through:

Setting up Mosquitto on a Raspberry Pi
Configuring the ESP32 to send sensor data via MQTT
Deploying a real-time web-based dashboard for visualization

1️⃣ Install Mosquitto on Raspberry Pi (Raspbian)

Run the following commands to install Mosquitto:

sudo apt update -y
sudo apt install mosquitto

2️⃣ Configure Mosquitto Broker

Edit the Mosquitto configuration file:

sudo vi /etc/mosquitto/mosquitto.conf

Copy and paste the following configuration:

pid_file /run/mosquitto/mosquitto.pid

persistence true
persistence_location /var/lib/mosquitto/

log_dest file /var/log/mosquitto/mosquitto.log

include_dir /etc/mosquitto/conf.d

allow_anonymous false  # Change to 'true' for public connections
password_file /etc/mosquitto/passwd

listener 1883 0.0.0.0  # Allow binding to the network

Save and exit, then restart Mosquitto:

sudo systemctl restart mosquitto

3️⃣ Enable and Start Mosquitto

sudo systemctl enable mosquitto
sudo systemctl start mosquitto

4️⃣ (Optional) Set a Username and Password

If allow_anonymous false is set, create a username and password:

sudo mosquitto_passwd -c /etc/mosquitto/passwd myUsername

Restart Mosquitto:

sudo systemctl restart mosquitto

5️⃣ Test the MQTT Server

Install Mosquitto Clients

sudo apt install mosquitto-clients

Publish a Message

Without authentication:

mosquitto_pub -h localhost -t "test/topic" -m "Hello, Raspberry Pi!"

With authentication:

mosquitto_pub -h localhost -u "myUser" -P "myPassword" -t "test/topic" -m "Hello, Raspberry Pi!"

Subscribe to a Topic

Without authentication:

mosquitto_sub -h localhost -t "test/topic"

With authentication:

mosquitto_sub -h localhost -u "myUser" -P "myPassword" -t "test/topic"

Tip: For better performance, set a static IP on your Raspberry Pi:

🔗 Guide: Set a Static IP on Raspberry Pi

6️⃣ ESP32 Circuit Setup

We'll use an ESP32 with an LM35 temperature sensor.

🔗 Code Repository: ESP32 MQTT Code

Set WiFi Credentials

Modify boot.py:

SSID = "Your_SSID"  # WiFi network name
PASSWORD = "Your_PASSWORD"  # WiFi password

Set MQTT Credentials

Modify main.py:

mqtt_server = "YOUR_RASPBERRY_IP"
mqtt_user = "myUser"  # Optional
mqtt_pass = "myPassword"  # Optional

Upload the Code to ESP32

ampy --port /dev/ttyUSB0 put boot.py
ampy --port /dev/ttyUSB0 put main.py
ampy --port /dev/ttyUSB0 put mqtt.py

Note: ttyUSB0 is the ESP32's serial port.

Circuit Diagram

Build the circuit as shown in the image:

D34 is used as the analog-to-digital converter (ADC).

7️⃣ Receiving Data from ESP32

Once everything is set up, the ESP32 will send data to the MQTT topic sensors in this format:

[{"name": "temperature", "value": 29.9}]

This happens inside main.py:

while True:
    try:
        client.check_msg()
        if (time.time() - last_message) > message_interval:
            sensors = [{"name": "temperature", "value": read_temp()}]  # Sensor data sent to MQTT
            client.publish(topic_pub, json.dumps(sensors))
            last_message = time.time()
        time.sleep(1)
    except OSError as e:
        print("Failed to connect to MQTT broker. Restarting...", e)
        restart_and_reconnect()

8️⃣ Sensor Dashboard Integration

Once the ESP32 sends data via MQTT, you can visualize it in real time using a web-based sensor dashboard with FastHTML

🔗 Dashboard Code Repository: Sensor Dashboard

Run the Sensor Dashboard

Clone the repository and install dependencies:

git clone https://github.com/BloodBoy21/sensorsDashboard.git
cd sensorsDashboard
pip install -r requirements.txt

Set MQTT Credentials in `.env`

MQTT_BROKER="MY_RASPBERRY_IP"
MQTT_USER=""
MQTT_PASSWORD=""

Run the Dashboard

python main.py

Access it in your browser:

http://localhost:5001

Alternatively, run using Docker:

docker compose up

9️⃣ Adding More Sensors

Modify main.py to add new sensors:

list_of_sensors = [
    Sensor(
        "temperature",  # Name from MQTT payload
        "Living Room",
        layout=Layout(
            title="Temperature Sensor",
            width=400,
            height=400,
        ),
        view_data=[
            View(
                title="Temperature",
                type="indicator",
                mode="number+gauge", # Type of chart
                domain={"x": [0, 1], "y": [0, 1]},
            )
        ],
    ),
    Sensor(
        "humidity",
        "Living Room",
        layout=Layout(
            title="Humidity Sensor",
            width=400,
            height=400,
        ),
        view_data=[
            View(
                title="Humidity",
                type="indicator",
                mode="number+gauge",
                domain={"x": [0, 1], "y": [0, 1]},
            )
        ],
    ),
    # Add more sensors here
]

my_dashboard.add_sensors(list_of_sensors)

🎉 Final Result: Live Sensor Dashboard!

Now, you can visualize real-time sensor data with interactive charts.

🔗 References

🚀 Now you have a complete IoT system with ESP32, MQTT, and a live sensor dashboard! 🚀

Deploy cloud function using CLI

Alan — Sun, 28 Jan 2024 22:02:40 +0000

Sometimes, deploying a FaaS (Function As a Service) becomes necessary to simplify our main monolith or when dealing with a microservices infrastructure.

In this blog, I will demonstrate how to deploy certain cases of FaaS in Google Cloud.

Http function

The most common scenario involves an HTTP invocation, and for a Node.js function, the syntax follows an Express-style.

exports.entry = (req,res) => {
  const {body,params,query} = req;
  const message = body?.message || params?.message || query?.message || 'Hello World';
  res.status(200).send(message);
}

We need to export a function with the same name as our declared entry point. This function is the one that GCP (Google Cloud Platform) should call on each run.

First, navigate to the root path of your project, and then run the following command:

gcloud functions deploy my-function --runtime nodejs20 --region us-west1 --source . --entry-point=entry --trigger-http --allow-unauthenticated --project=my-dev-project-412620

We are instructing the CLI to deploy a function named 'my-function,' utilizing Node.js v20 as its runtime. The function has an entry point named 'entry' and is accessible via HTTP in the us-west1 region (Oregon). It can be invoked without a GCP IAM token (use --no-allow-unauthenticated to disable this). This resource will be deployed in 'my-dev-project-412620'.

After running this command, if the CLI displays a message such as 'API [cloudfunctions.googleapis.com] not enabled on project [my-dev-project-412620]. Would you like to enable and retry (this will take a few minutes)? (y/N)?' simply type 'y' to proceed.

Now, navigate to gcloud -> functions, and you should be able to view our newly deployed function.

Now, you can make calls using the Cloud Function URL

Tips

SQL connections

Cloud functions are serverless. If you need to run a function with a connection to a database, you may encounter a pool connection timeout error. To prevent this error, it is a good practice to set the connection pool limit to 1 and the pool timeout to 0.

Fast function

Cloud functions reuse existing containers to handle new calls. Following this logic, a good practice is to separate one-use code from the request-logic code. Define the one-use code outside of the main function and keep only the request-handling logic inside the specified entry-point function.

const {connect_to_db} = require('./db');

connect_to_db()
// connections and another code here
exports.entry = (req,res) => {
  // request handler logic here
  const {body,params,query} = req;
  const message = body?.message || params?.message || query?.message || 'Hello World';
  res.status(200).send(message);
}

Flags

--gen2

By using this flag, you can enable the second-generation function and deploy a Cloud Run as a function. Additionally, you can define the CPU and memory usage using the --cpu and --memory flags.

--env-vars-file

With this flag, you can define an environment file in YAML format to pass variables to the Cloud Function.

DATABASE_URL: <MY_URL>
PORT: 3000

--vpc-connector

This flag is used to designate the VPC Connector for a Cloud Function. It allows you to specify the VPC network that your function will be connected to, providing control over networking configurations

Resource

CLI documentation

You know some tricks about cloud functions and serverless structure?

Use gRPC inside monolith

Alan — Fri, 29 Sep 2023 07:18:50 +0000

Many times, we start a project with a monolithic architecture. However, over time, as the project scales, it may become necessary to transition to a microservices architecture. But what if your code is already used in other parts of your application? In this article, I will show you a way to implement gRPC in your monolithic architecture.

Monolith

The first step is create your main monolith server with necessary logic
server.js

const express = require("express");
const app = express();
const morgan = require("morgan");
const bodyParser = require("body-parser");
const { legacyController } = require("./controllers");

const PORT = process.env.PORT || 3001;

app.use(morgan("dev"));
app.use(bodyParser.json());
app.use(bodyParser.urlencoded({ extended: true }));

app.get("/", (req, res) => {
  res.send("Hello World");
});

app.get("/api/users", async (req, res) => {
  const { sort = "desc" } = req.query;
  const users = await legacyController.getUsers(sort);
  return res.json(users);
});

const server = app.listen(PORT, () => {
  console.log(
    `Server is running on port ${server.address().port}\nhttp://localhost:${
      server.address().port
    }`
  );
});

controllers/legacyController.js

const getUsers = async (sort) => {
  const users = [
    { name: "John", age: 21 },
    { name: "Jane", age: 22 },
  ];
  if (sort === "desc")
    return Promise.resolve(users.sort((a, b) => b.age - a.age));

  return Promise.resolve(users.sort((a, b) => a.age - b.age));
};

module.exports = {
  getUsers,
};

controllers/index.js

module.exports = {
  legacyController: require("./legacyController")
}

The response to GET https://localhost:3000/api/user is:

[
    {
        "name": "Jane",
        "age": 22
    },
    {
        "name": "John",
        "age": 21
    }
]

This represents the legacy logic, and now we're going to implement it in our new gRPC server as a microservice.

gRPC server

server.js

const grpc = require("@grpc/grpc-js");
const { grpc: legacyController } = require("./legacyController/infrastructure");

function main() {
  const PORT = process.env.PORT || 50051;
  const server = new grpc.Server();
  const services = [[legacyController.service, legacyController.methods]];
  services.forEach(([service, methods]) => {
    server.addService(service, methods);
  });
  const serverCredentials = grpc.ServerCredentials.createInsecure();
  server.bindAsync(`0.0.0.0:${PORT}`, serverCredentials, (error) => {
    if (error) throw error;
    console.log(`Server running at http://localhost:${PORT}`);
    server.start();
  });
}
main();

legacyController/application/index.js
In this part, it's important to use the same package name as specified in the model.proto file, and remember that package names are case-sensitive.

const path = require("path");
const protoPath = path.join(__dirname, "..", "domain", "model.proto");
const grpc = require("@grpc/grpc-js");
const protoLoader = require("@grpc/proto-loader");
const methods = require("./senders");
const packageDefinition = protoLoader.loadSync(protoPath, {
  keepCase: true,
  longs: String,
  enums: String,
  defaults: true,
  oneofs: true,
});
const service = grpc.loadPackageDefinition(packageDefinition).legacyController;

module.exports = {
  methods,
  service: service.Output.service,
};

legacyController/application/methods.js
In this file, we utilize the same logic as in our legacy controller, with the distinction that function parameters are passed in an object.

async function getUsers({ sort }) {
  const users = [
    { name: "John", age: 21 },
    { name: "Jane", age: 22 },
  ];
  let data = [];
  if (sort === "desc") data = users.sort((a, b) => b.age - a.age);
  else data = users.sort((a, b) => a.age - b.age);
  return {
    parse: true,
    data: JSON.stringify(data),
  };
}
module.exports = {
  getUsers,
};

legacyController/application/senders.js

In this function, we bind our main function written in methods. However, gRPC operates using callbacks, so we employ this implementation to ensure seamless interaction with asynchronous functions and proper data return

const methods = require("./methods");

// Function to be used as grpc method pass first to main function and then return data to callback response
async function functionParser({ request }, callback) {
  const { mainFunction } = this;
  mainFunction(request).then((data) => callback(null, data));
}

// Create object with all functions from methods and bind them to functionParser to be used as grpc methods
const senders = Object.entries(methods).reduce(
  (acc, [functionName, functionVal]) => {
    acc[functionName] = functionParser.bind({
      mainFunction: functionVal,
    });
    return acc;
  },
  {}
);

module.exports = { ...senders };

legacyController/domain/model.proto
In this model, we write function parameters and return types using hard typing. It's crucial to use the same function names in both the model and client implementations.

We define data as a string because if we need to pass an array with an unknown type structure, it would be challenging to implement it as a message structure in the proto model.

syntax = "proto3";

package legacyController;
import "google/protobuf/any.proto";
service Output {
  rpc getUsers (UserParams) returns (Response) {}
}

message UserParams {
  string sort = 1;
}

message Response {
  string data = 1;
  bool parse = 2;
}

legacyController/infrastructure/index.js

Finally, we export all the necessary methods to use in server.js.

const { methods, service } = require("../application");

module.exports = {
  grpc: {
    service,
    methods,
  },
};

Update legacy code

First, we need to save the proto file and ensure it's updated on both sides whenever changes are made.

protos/legacyController.proto

syntax = "proto3";

package legacyController;
import "google/protobuf/any.proto";
service Output {
  rpc getUsers (UserParams) returns (Response) {}
}

message UserParams {
  string sort = 1;
}

message Response {
  string data = 1;
  bool parse = 2;
}

Now, we return to the monolith server and head to controllers/legacyController.js. Here, we'll create the gRPC client to establish a connection with the other server.

const { getGrpcClient } = require("../helpers");
const CLIENT_NAME = "legacyController";
const path = require("path");
const protoPath = path.join(
  __dirname,
  "..",
  "protos",
  "legacyController.proto"
);
const client = getGrpcClient(protoPath, CLIENT_NAME);

helpers/grpcClient.js

const grpc = require("@grpc/grpc-js");
const protoLoader = require("@grpc/proto-loader");
const LOCAL_HOST = "localhost:50051";

const loadProto = (protoPath) =>
  protoLoader.loadSync(protoPath, {
    keepCase: true,
    longs: String,
    enums: String,
    defaults: true,
    oneofs: true,
  });

const getGrpcClient = (protoPath, serviceName, host = LOCAL_HOST) => {
  const packageDefinition = loadProto(protoPath);
  const service = grpc.loadPackageDefinition(packageDefinition)[serviceName];
  const client = new service.Output(host, grpc.credentials.createInsecure());
  return client;
};

module.exports = getGrpcClient;

helpers/index.js

module.exports = {
  getGrpcClient: require("./grpcClient"),
};

Comment the old code

/* const getUsers = async (sort) => {
const users = [
  { name: "John", age: 21 },
  { name: "Jane", age: 22 },
];
if (sort === "desc")
  return Promise.resolve(users.sort((a, b) => b.age - a.age));

return Promise.resolve(users.sort((a, b) => a.age - b.age));
}; */

We create a function to pass arguments by function to object

const transformMethodParams = {
  getUsers: ["sort"],
};

const passArgsToObj = (args, method) => {
  const params = transformMethodParams[method];
  const context = {};
  if (!params) return context;
  params.forEach((param, index) => {
    context[param] = args[index];
  });
  return context;
};

Since gRPC operates using callbacks and our code is asynchronous, we utilize an abstraction to intercept all calls to our function and wait for them to complete or handle errors when they occur.

async function transformCall(...args) {
  const { functionName } = this;
  const context = passArgsToObj(args, functionName);
  return new Promise((resolve, reject) => {
    client[functionName](context, {}, (err, response) => {
      if (err) return reject(err);
      if (response?.parse) return resolve(JSON.parse(response?.data));
      resolve(response.data);
    });
  });
}

const outputMethod = (method) => transformCall.bind({ functionName: method });

module.exports = {
  getUsers: outputMethod("getUsers"),
};

full code

const { getGrpcClient } = require("../helpers");
const CLIENT_NAME = "legacyController";
const path = require("path");
const protoPath = path.join(
  __dirname,
  "..",
  "protos",
  "legacyController.proto"
);
const client = getGrpcClient(protoPath, CLIENT_NAME);

/* const getUsers = async (sort) => {
const users = [
  { name: "John", age: 21 },
  { name: "Jane", age: 22 },
];
if (sort === "desc")
  return Promise.resolve(users.sort((a, b) => b.age - a.age));

return Promise.resolve(users.sort((a, b) => a.age - b.age));
}; */

const transformMethodParams = {
  getUsers: ["sort"],
};

const passArgsToObj = (args, method) => {
  const params = transformMethodParams[method];
  const context = {};
  if (!params) return context;
  params.forEach((param, index) => {
    context[param] = args[index];
  });
  return context;
};

async function transformCall(...args) {
  const { functionName } = this;
  const context = passArgsToObj(args, functionName);
  return new Promise((resolve, reject) => {
    client[functionName](context, {}, (err, response) => {
      if (err) return reject(err);
      if (response?.parse) return resolve(JSON.parse(response?.data));
      resolve(response.data);
    });
  });
}

const outputMethod = (method) => transformCall.bind({ functionName: method });

module.exports = {
  getUsers: outputMethod("getUsers"),
};

Conclusions

In conclusion, gRPC is a powerful protocol for microservices communication, offering strong typing to ensure the integrity of data transmitted between clients and servers. Its implementation in monolithic systems allows for a strategic deconstruction of the monolith into smaller, more manageable parts.

Repository