DEV Community: Jonathan Ato Markin

FastMap: Real-Time IoT Anomaly Detection with Redis's Multi-Model Database

Jonathan Ato Markin — Sun, 10 Aug 2025 12:07:32 +0000

This is a submission for the Redis AI Challenge: Beyond the Cache.

What I Built

In today's connected world, monitoring critical infrastructure like water systems, power grids, or server farms is a monumental task. A single failure can be catastrophic. The challenge is detecting problems the instant they occur, not minutes later when a traditional database finally runs its report.

I built FastMap, a real-time anomaly detection platform for large-scale sensor networks. It provides a live map-based dashboard where operators can monitor thousands of IoT sensors at a glance. When a sensor reports an anomalous value, like a sudden pressure spike or temperature drop, FastMap identifies it in milliseconds and instantly flags the sensor on the map, allowing for immediate intervention.

The entire application is powered exclusively by Redis, which acts as the high-speed data ingestion layer, the real-time AI/ML feature store, the multi-model primary database, and the real-time notification engine all in one.

Demo

Here is a live demonstration of FastMap in action. You can see the real-time updates as sensors report data and anomalies are triggered.

Video Demo: FastMap Demo Video
Source Code: FastMap Redis Github

Below is also a screenshot of the user interface

How I Used Redis 8

For FastMap, Redis wasn't just a cache; it was the single source of truth and the engine for the entire application. I combined multiple Redis data models to build a robust system without needing a separate relational or NoSQL database.

Primary Database:
- Redis Streams: The front door for all incoming data, providing a durable, scalable log for our high-throughput sensor network.
- RedisJSON: Used to store rich, structured metadata for each sensor, including its location, type, and operational parameters. Querying this data was flexible and intuitive.
- Redis TimeSeries: To store historical readings for every sensor, allowing for efficient time-range queries, aggregations, and future trend analysis—all natively within Redis.
- Redis Hashes: For storing the volatile, high-access "current state" of each sensor (e.g., status: 'ANOMALY'), enabling lightning-fast lookups for the dashboard.
Real-Time Messaging:
- Redis Pub/Sub: When an anomaly was detected, a message was published to a Pub/Sub channel. This decoupled the backend logic from the frontend dashboard, allowing us to broadcast alerts to all connected users instantly with minimal overhead.

This multi-model approach, all within a single Redis instance, demonstrates its incredible power and versatility as a complete, real-time data platform.

Local Development, Remote Data: Accessing Fly.io PostgreSQL from Your Java API

Jonathan Ato Markin — Sun, 26 Jan 2025 20:12:32 +0000

Introduction

Inter-communication between databases when developing applications is not an entirely new challenge to developers or the software engineering space. I recently found myself in a situation where I had to access data from an external PostgreSQL database while developing a Java API locally. The external database was hosted on Fly.io and my local development setup had to interact with it seamlessly. After some research and consultation, I discovered that I could harness the power of PostgreSQL Foreign Data Wrapper (FDW) and Fly Proxy to make it work. In this article, I’ll walk you through my journey, the steps I took, and how you can apply this solution to your projects.

The Challenge

I was working on a Java API(with SpringBoot) that needed to fetch data from a table that was being managed by another application's Postgres database. The external application and its database were all hosted on Fly.io. While developing locally, I could not directly connect to the Fly Postgres database due to network restrictions.

I had two main issues to contend with

Establishing a connection or communication channel with Fly.io from my local setup
Accessing the data from the external database on Fly.io

Finding solutions for the above issues led me to discover Fly Proxy and PostgreSQL Foreign Data Wrapper.

What Is PostgreSQL Foreign Data Wrapper?

PostgreSQL Foreign Data Wrapper (FDW) is a powerful module that allows you to query external data sources as if they were local. It acts as a bridge between the local database and the external one.

I used the postgres_fdw extension to connect to the external Fly.io PostgreSQL database.

Setting Up The Proxy

You should install flyctl before proceeding to carry out these procedures.
Since Fly.io restricts direct access to its databases, I needed a way to securely connect to it from my local machine. I used Fly.io’s flyctl proxy to create a secure tunnel. The general command is given by:

fly proxy <local:remote> [remote_host] [flags]

In setting up my proxy I needed to know the name of the remote_host which was the Fly Postgres database application I wanted to interact with. I then set up the proxy with a similar command as below:

fly proxy 5433:5432 -a db_app_name

This created a local tunnel to the Fly.io database. With my proxy set up, I proceeded to my FDW to interact with the external database.

Setting Up PostgreSQL FDW

Step 1: Install the postgres_fdw Extension

First, I installed the postgres_fdw extension in my local PostgreSQL database:

CREATE EXTENSION IF NOT EXISTS postgres_fdw;

Step 2: Create a Foreign Server Object

Next, I defined the connection to the external database by creating a foreign server object to represent the remote database I wanted to connect to.

CREATE SERVER fly_server FOREIGN DATA WRAPPER postgres_fdw
OPTIONS (
    host '127.0.0.1',
    port '5433',
    dbname 'db_name'
);

fly_server can be replaced with any preferred name.

Step 3: Map a Local User to the Remote User

I created a user mapping for each database user I wanted to allow to access each foreign server. In my case, I did for only one user.

CREATE USER MAPPING FOR local_user SERVER fly_server
OPTIONS (
    user 'remote_user',
    password 'remote_password'
);

Step 4: Link the External Table

Next, I had to create a foreign table for each remote table I wanted to access.
I used the IMPORT FOREIGN SCHEMA approach and limited it to the table I was interested in.

IMPORT FOREIGN SCHEMA foreign_schema_name
LIMIT TO (remote_table)
FROM SERVER fly_server
INTO local_schema_name;

In my case as well, there was an ENUM TYPE in the foreign database that the table depended on so I had to create that TYPE as well in my local database.

Step 5: Create Materialized View (Optional)

This is an optional step that I found to be useful, especially in situations where data is seldom updated and the only operations being performed are READ operations.

Integrating with Your Java API

With the FDW and proxy in place, I could now query the external data directly from my SpringBoot application using either my materialized view or the imported foreign table directly by establishing a connection with my local database.

Lessons Learned

FDW is powerful but requires careful setup to get things working
flyctl proxies make it easy to create secure tunnels to interact with remote databases
This approach also enabled me to treat my app as a separate service with its own db decoupled from the remote application even though it depended on some data from it.

Conclusion

Using PostgreSQL Foreign Data Wrappers and a flyctl proxy, I was able to seamlessly integrate external data into my Java web application. This approach not only solved my immediate problem but also opened up new possibilities for working with distributed data.
If you’re facing a similar challenge, I highly recommend giving FDW a try!

Feel free to connect with me on LinkedIn.

System.out.println("Happy Coding!🚀")

Spring Boot Basics: Crafting Your First Application

Jonathan Ato Markin — Sat, 24 Jun 2023 18:57:46 +0000

🚀 Introduction

Web application development stands at the core of building impactful solutions in almost every field and organization.
Among the many languages, Java continues to stand tall and showcase its usefulness and longevity, having been ranked 3rd on the Top Programming Languages at the 2022 Github Octoverse.
Spring Boot, currently the most popular framework in the Java ecosystem, offers a simplified and versatile approach to building web applications. This article aims to provide a simplified roadmap for writing your first Spring Boot Java application.

📝 Prerequisites

Before starting with this tutorial, there are a few prerequisites you should take care of:

Java Development Kit (JDK): Spring Boot 2.4.0 and later requires at least Java 8. You can download the latest version of the JDK from Oracle's official website.
Integrated Development Environment (IDE): A good IDE makes the development of Spring Boot applications simpler and easier. I recommend downloading and using IntelliJ IDEA but you are free to use any IDE of your choice like Apache NetBeans, Eclipseor VSCode.
Basic Java knowledge: This tutorial assumes that you have a basic understanding of Java programming. You don't need to be an expert, but you should understand Java syntax and simple concepts like classes, objects, and methods. If you are new to Java, feel free to check out this Java For Beginners Series by Microsoft, the official Java learn website or any trusted source.
Familiarity with Maven or Gradle: Spring Boot uses these tools to manage dependencies. While it's not essential to have in-depth knowledge, a basic understanding of how these tools work will be useful. In this tutorial, we will use Maven.

🛠️ Create A New Spring Boot Application

Spring Initializr is a tool provided to help create Spring Boot applications easily.

Visit https://start.spring.io/. You should see something similar to the image below.
Select Java and Maven under the Project and Language sections respectively.
Select the Spring Boot version of your choice or leave it at the recommended version.
Name your project as you desire under the Project Metadata and choose a Java version as well.
In the Dependencies section, add Spring Web to create a web application.
Finally, click on the Generate button. This will download a .zip file containing your new Spring Boot project.

🧐 Exploring The Project Structure

Unzip the downloaded file and open the project in IntelliJ IDEA or your favourite IDE. Your project structure should look like this.

📂 Understanding The Project Structure

src: This directory contains all the source code for the project.
- main: This folder houses the primary source code for the application.
  - java: This is where all our Java source files reside.
    - com/example/demo: This is the package. .
      - DemoApplication.java: This is the main entry point for the Spring Boot application. Spring Boot applications bootstrap from a main method, which is contained in this class.
  - resources: This folder contains all non-code resources, like JSON, XML, properties files, and static web resources. It's also where we keep:
    - application.properties: This file is where you can specify application configurations, like server port, database connection info, etc.
- test: This folder contains all of our test source code.
  - java: Here is where all our Java test files reside.
    - com/example/demo: This follows the same package structure as the main codebase.
      - DemoApplicationTests.java: This is where we write tests for our application.
.gitignore: This file tells Git which files or directories to skip for version control.
pom.xml: If you're using Maven as your build tool, this is where your project's dependencies and plugins are managed. In our case, we are using Maven so our dependences will reside here.
README.md: This file contains documentation of the project – what it does, how to set it up, and other useful information.

✍️ Writing Your First Controller

Controllers in Spring Boot handle incoming web requests and return responses. Let's create a simple controller.

Create a new file called FirstController.java in the same package as DemoApplication.java. You can name the controller anything you want. Paste this code in your controller:

package com.example.demo;

import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RestController;

@RestController
public class FirstController {

    @GetMapping("/first")
    public String first() {
        return "My First Spring Boot App!";
    }
}

📖 Code Explanation

@RestController: This is an annotation that is itself annotated with @Controller and @ResponseBody. @Controller is an annotation that marks the class as a web controller, capable of handling incoming HTTP requests. @ResponseBody indicates that the return values from the methods in this class will be bound to the HTTP response body.
@GetMapping("/first"): This annotation is used to map HTTP GET requests onto specific handler methods. Here, it's mapping an HTTP GET request to the /first path to the first() method. When someone navigates to /first on your web service, the first(){...} method will be invoked.
You can read more about Spring Annotations via the Spring official reference documentation or access a cheat sheet here

🏃 Running Your Application

You can run your application easily from your IDE by clicking the RUN button of your IDE, normally shaped as a Play button.
You can also run the application from the terminal using the following command from the project root directory: ./mvnw spring-boot:run
Open your favorite browser and navigate to http://localhost:8080/first. You should see the message "My First Spring Boot App!".

Note: By default, Spring Boot applications run on port 8080. If you already have another application running on this port, you will encounter a port conflict, meaning your Spring Boot application won't start correctly. To resolve this, you can configure your Spring Boot application to use a different port. You can do this in the application.properties file like so:

server.port=9090

With this configuration, your application will run on port 9090 instead of the default port 8080.

🎉 Congratulations! 🎉

You've just developed your first Spring Boot Java application! This is merely the initial step in your journey with Spring Boot. The Spring Boot framework offers many other features 🚀 that you can delve into as you progress in your career as a developer.

Remember to practice consistently and always challenge yourself!
Happy coding! 🎊👩‍💻🎊