Building a #Growlab using Raspberry Pi & Waylay

Sander Vanhove — Tue, 25 May 2021 15:20:23 +0000

Intro

Last month Alex Ellis was the lone nut who started the #Growlab project. His idea was simple: grow your own vegetables while learning something new technology-wise.

When he showed me the project I really wanted to participate, so I bought some sensors, dusted of my old Raspberry Pi and got to creating my setup.

Hardware

My final setup looks like this:

// Detect dark theme var iframe = document.getElementById('tweet-1391665619616026624-923'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1391665619616026624&theme=dark" }

I'm using a:

Raspberry Pi 3b+, but you could definitely run everything on a less powerful device like a Raspberry Zero.
Raspberry Camera v2
Grove Moisture sensor
Grove Temperature/Humidity sensor
Grove Light sensor
Grove 16x2 Display
Grove Base Hat to connect all the Grove components

Everything is build into an IKEA rack I had standing around. I had to find a home for the stuff it was housing, but it's a nice frame. Then I just cut some wood to the right dimensions and zip-tied everything together.

It may not be the most beautiful setup, but like Alex said in his talk about Growlab: "This is about making due".

No matter what hardware or sensors you are using, the growlab project is about creating and learning something and not so much about exactly copying what others are doing.

Software

I'm only going to cover the link to Waylay here, else this article would get way to long. But you are invited to check out my complete code solution on this Github repository.

Measurements

Getting measurements from the sensors is very manufacturer specific. The Grove system has its own grove.py python library that I am using in my sensors.py code to return an object with this signature:

@dataclass
class SensorReading:
    light: float
    temperature: float
    humidity: float
    moisture: float

Posting to Waylay

Getting these measurement values into Waylay is not that hard from here on. The steps we want to take are:

Create a Resource
Create a Webscript
Send data to the Webscript

1. Create a Resource

Resources are the base of the Waylay system, they represent physical devices or groups of them. Check out this video tutorial to get a better grasp on what they are. Let's create a new one and name it plant-monitor:

Copy that Identifier, we'll need it in the next step.

2. Create a Webscripts

Webscripts are lambda functions within the Waylay system. We first send the data to one of these because it is able to process the data before storing it to a resource. Now when I want to change some processing of the data (for example dividing the humidity by 100 so it is a value from 0 to 1 instead of 0 to 100), I can change the webscript in a nice UI environment instead of having to ssh into my Raspberry Pi.

I will be sending to a Webscript directly because that is easiest way to get started. If you want to build out a bigger use case or to run something in production, MQTT is probably the better option. A Webscript can still be used after data is ingested by MQTT.

Create a new Webscript in Waylay and paste this code, or something similar:

async function handleRequest (req, res) {
  if (!req.body) {
    // No body found
    return
  }

  // Parse body if needed
  let payload = req.body
  if (typeof payload === 'string' || payload instanceof String)
    payload = JSON.parse(payload)

  /*
    You can do some processing of the payload over here.
  */

  // Post values to our resource
  waylay.data.baseUrl = 'https://data-io.waylay.io'
  await waylay.data.postSeries('resource-id', payload, { store: true, forward: true })
    .catch(e => console.error(e.message))

  res.sendStatus(200)
}

Don't forget to replace 'resource-id' with the Identifier of your resource.

Bonus points: Send the id of your resource along in the payload. This way you can have multiple plant monitors running.

3. Send data to the Webscript

Now all we have to do is send the metric values from our Raspberry Pi to that Webscript. First copy the Webscript's link by clicking on the URL underneath its name:

If your Webscript is private (which it is by default), this copied URL also includes the secret token to authenticate, so we don't have to do any more auth in the code.

We can use this URL to POST values to on our Raspberry like this:

from requests import post
from dataclasses import asdict

json_data = asdict(data)
request = post("webscript-url", json=json_data)

Again change the "webscript-url" with your URL. Check out the webscript.py file in my repo to see my full implementation that also does some error handling.

I glossed over a bunch of things about taking pictures and storing them to Github, so please check out the Github repo if you want a more indepth view of my code. But these are the basic steps to get values into the Waylay system.

Viewing the data

Now that everything is set up we can start pushing data from the Raspberry Pi. We should be able to see this data coming in under the data tab of our resource:

If you press Explore next to one of the metrics, you are taken to the explorer view. You can learn more about the explorer view in this video tutorial. If you add the other metrics too, you have an overview of everything at once:

Huray, we did it! 🥳

Dashboard

Lastly I also created a little dashboard that quickly gives me the stats of my plants. As this article is already getting quite long I'll leave explaining this for another time.

// Detect dark theme var iframe = document.getElementById('tweet-1394224807013666816-164'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1394224807013666816&theme=dark" }

Further improvements

There are so many ways to keep improving this project! I'll list a few here:

Automations

There are no automations being done for the moment. I just have the monitoring and dashboard set up. I could have some automated alarms firing when conditions are not ideal for the little plants.

Closing the loop

At the moment I'm only monitor the plant, but it would be so much more interesting if I could close the loop and also water the plant using a little pump. Then I need a sensor that measures the water level in the reservoir too 🤔

MQTT

Using a Webscript directly is easy to get started with, but using MQTT is the way to go in production environments.

ML

You can do a lot of cool things with Machine Learning, some cool ideas that could be incorporated here:

Germination detection: detect whether the seeds have germinated so I can get a notification when they do.
Bad leaf detection: determine if there are leafs going bad. This dataset could be used.
Droop detection: see if the leafs are drooping, which is generally a bad sign.

More cool ML applications in agriculture can be found in this awesome paper.

Conclusion

I certainly had a lot of fun putting this together and I hope my plants will be able to survive this time 😅

Again, this project is not about exactly copying what others are doing, but making due with what you have and just starting to experiment!

Hopefully this article got you on board with the #Growlab project. You now certainly have a jumping off point for growing your own vegetables and learning some tech-skills on the way.

What does a Data Orchestration platform do?

Sander Vanhove — Tue, 20 Apr 2021 10:11:53 +0000

Data Orchestration is becoming one of those buzzwords sales people like to throw around in pitch meetings.

It is very important to understand that data orchestration is not just one "thing", but describes a whole process of how to handle data within a system.

I identified 4 steps each data orchestration platform should support:

1. Data Ingestion

Data has to get into the system one way or another. We all know how it goes: part of the data we need is on an SQL database somewhere, other data comes in via MQTT and the last part is only available in .txt logs. Data ingestion can be a mess.

The first thing a Data Orchestration platform should be able to do is integrate with all these different sources so it is able to combine the data we need.

2. Data Processing

Data is rarely in the right format when entering a system. Therefore we need a way for data to be combined, unpacked or decoded. This step could also entail aggregation if we don't want to use all the data but just a higher order abstraction.

3. Making Decisions

Now that the data is ready, it is time to make some decisions based on what we ingested. For this, the platform needs some way to create rules and automated workflows. The cool platforms even have the ability to utilize AI or ML models and let those call the shots.

4. Taking Actions

Lastly it is time to take action! Based on the data and earlier decisions, the platform should be able to take actions in and outside of its own ecosystem. This could go from throwing alarms to turning some physical device on/off or sending quarterly figures to a certain list of employees.

The best of the best data orchestration platforms are flexible in each of these steps, making it a generally applicable solution. They let the developer integrate the system easily with other platforms and provides a way to quickly define new logic. But that is a topic for a whole other article.

Want an example of a data orchestration platform that does these things right? Check out Waylay IO.

DEV Community: Sander Vanhove