ELTAI: web-based alternative to Apache Airflow for MLOps and AI

#redischallenge #devchallenge #database #ai

Redis AI Challenge: Real-Time AI Innovators

This is a submission for the Redis AI Challenge: Real-Time AI Innovators.

What I Built

ELTAI is a web-based service that can scrape the contents of any webpage as source of data for AI, even password-protected ones, however this use of it is not advisable since it is still a budding project.

An Apache Airflow alternative sounds quite ambitious but that is the future of this project

Overview

EXTRACT: Downloads and renders the markup returned by the provided URL as a visual guide, from which support using a custom script to extract text is provided

LOAD: A database URL is provided to read or write data to

TRANSFORM: Use custom scripts to modify data before writing it to database or vectorizing it into embeddings

AI:(in development) Semantic search, filtering and CI/CD whereby filtered data is sent to a specified remote server over a specified period

Demo

Demo: https://eltai.pages.dev/

Repo: https://github.com/ogbotemi-2000/ELTAI

How I Used Redis 8

Interactive database session

Database queries are sent to index function on Vercel with multipart/form-data encoding to be written or read. Success or errors are displayed and explained

Custom AI-powered operations

The embedding generated for the provided and transformed data are received from the Cloudflare worker used to generate them to be stored in Redis and used along with custom specifications by user for any vector operation: semantic search, filtering.

The data passed through Redis vector databases an operations along with embedding generated from the AI models or the default Cloudflare AI models is then sent in a timely manner specified prior to work as a workflow suitable for devs who want a large data source - the internet.