DEV Community: Ogbotemi Ogungbamila

ELTAI: web-based alternative to Apache Airflow for MLOps and AI

Ogbotemi Ogungbamila — Mon, 11 Aug 2025 04:21:19 +0000

This is a submission for the Redis AI Challenge: Real-Time AI Innovators.

What I Built

ELTAI is a web-based service that can scrape the contents of any webpage as source of data for AI, even password-protected ones, however this use of it is not advisable since it is still a budding project.

An Apache Airflow alternative sounds quite ambitious but that is the future of this project

Overview

EXTRACT: Downloads and renders the markup returned by the provided URL as a visual guide, from which support using a custom script to extract text is provided

LOAD: A database URL is provided to read or write data to

TRANSFORM: Use custom scripts to modify data before writing it to database or vectorizing it into embeddings

AI:(in development) Semantic search, filtering and CI/CD whereby filtered data is sent to a specified remote server over a specified period

Demo

Demo: https://eltai.pages.dev/

Repo: https://github.com/ogbotemi-2000/ELTAI

How I Used Redis 8

Interactive database session

Database queries are sent to index function on Vercel with multipart/form-data encoding to be written or read. Success or errors are displayed and explained

Custom AI-powered operations

The embedding generated for the provided and transformed data are received from the Cloudflare worker used to generate them to be stored in Redis and used along with custom specifications by user for any vector operation: semantic search, filtering.

The data passed through Redis vector databases an operations along with embedding generated from the AI models or the default Cloudflare AI models is then sent in a timely manner specified prior to work as a workflow suitable for devs who want a large data source - the internet.

User information storage

All user data will be stored in Redis

AI-pipe: Pipeline for generating/storing embeddings from AI models to DB with data scraped from sites using custom scripts

Ogbotemi Ogungbamila — Mon, 30 Dec 2024 07:53:18 +0000

This is a submission for the Bright Data Web Scraping Challenge: Most Creative Use of Web Data for AI Models

What I Built

A web page to quickly create a pipeline to feed AI models data scraped from a provided webpage.

Features

Custom scriptinig

Total control over the kind, type and form of data scraped from webpages is given in the form of custom scripts with templates provided.

Embeddings generation

The web service supports generating embeddings from OpenAI and Ollama AI models. It also provides a fallback for users without access to AI models running on a remote server through PostgresML

Demo

Coming in rather late but here is a link to a deployed demo of the webapp below

https://ai-pipe.vercel.app/

ogbotemi-2000 / ai-pipe

A webapp that scrapes data you tell it to from the internet and lets you cleanse and format it which is then fed to an AI model to generate embeddings

ai-pipe

A webapp that scrapes data you tell it to from the internet and lets you cleanse and format it which is then fed to an AI model to generate embeddings

AI model providers

Ollama

Via a remote deployment like Koyeb

Open AI

Support for adding the API key along with the request body

PostgresML

Used as a fallback

Scraping data

You provide a URL and specify what nodes to target as well as what kind of data to extract from them all of which gets sent to the backend. The response is sent and you can work on each response for the nodes targeted by writings scripts to format, cleanse the data and preview the result before generating embeddings for it via an AI model of your choosing.

Embedding

The generated embedding is provided to be copied

In Addition

The webpage features links to useful resources on AI and…

View on GitHub

How I Used Bright Data

Scraping browser

I used Puppeteer along with a web socket URL that points to a browser provided by BrightData to access websites, mutate the DOM and traverse the DOM while applying custom scripts to scrape data from it.

Here is the code that handles the above

const puppeteer = require('puppeteer-core'),
      path   = require('path'),
      fs     = require('fs'),
      both = require('../js/both'),
      file   = path.join(require('../utils/rootDir')(__dirname), './config.json'),
      config = fs.existsSync(file)&&require(file)||{...process.env};

module.exports = function(request, response) {
  let { data } = request.body, result;
  let { nodes, url } = data = JSON.parse(data),
  /**serialize the needed function in the imported object for usage in puppeteer */
      _both = { asText: both.asText.toString() };

  new Promise(async (res, rej) => {  
    puppeteer.connect({
      headless: false,
      browserWSEndpoint: config.BROWSER_WS,
    }).then(browser=>browser.newPage().then(async page=>{

      await page.setUserAgent('5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36');
      await page.goto(url, { waitUntil:'load', timeout:1e3*60 });
      // await page.waitForFunction(() => document.readyState === 'complete');
      // await page.screenshot({path: 'example.png'});


      const result = await page.evaluate((nodes, both) => {
        /** convert serialized function string back into a function to execute it */
        both.asText = new Function(`return ${both.asText}`)()
        /**remove needless nodes from the DOM */
        document.head.remove(), ['link', 'script', 'style', 'svg'].forEach(tag=>document.body.querySelectorAll(tag).forEach(el=>el.remove()))
        /**defined "node" - the variable present in the dynamic scripts locally to make it available in the 
          custom function context when created with new Function */
        let page = {}, node, fxns = Object.keys(nodes).map(key=>
          /**slip in the local variable - page and prepend a return keyword to make the function string work 
           * as expected when made into a function
          */
          nodes[key] = new Function(`return ${nodes[key].replace(',', ', page, ')}`)()
        );
        /** apply the functions for the nodes to retrieve data as the DOM is being traversed */
        both.asText(document.body, (_node, res)=>fxns.find(fxn=>res=fxn(node=_node, page)) && /*handle fetching media assets later here*/res || '');
        return page
      }, nodes, _both);
      res(result), await browser.close();
    }).catch(rej))
    .catch(rej)
  }).then(page=>result = page)
  .catch((err, str, name)=>{
    str = err.toString(), name = err.constructor.name, result = {
      error: /^\[/.test(str) ? `${name}: A sudden network disconnect occured` : str
    }
  }).finally( ()=> {
    response.json(result)
  })
}

Web Unlocker

For stubborn sites that used Cloudflare Trunstile to prevent scraping, I tested some code using BrightData's proxy API and it worked!
In the future, I will implement a workaround whereby the downloaded HTML of the stubborn sites gets sent to the client-side to be scraped via scripts based on how useful people find this service.

Qualified Prompts

AI pipeline

My submission is primarily focused on this prompt however it happens to offer solutions to businesses that have always wanted to control and format the data they scrape from sites.

Thanks for reading

I built this for the BrightData challenge but I will improve on it if it turns out to be something useful

SwitchFlow: use your DevCycle account variables as UI toggles in prototyping. Export or download the results

Ogbotemi Ogungbamila — Mon, 23 Dec 2024 06:51:09 +0000

This is a submission for the DevCycle Feature Flag Challenge: Feature Flag Funhouse

What I Built

This is something I am happy to have built and, whether I win or not, I intend to make it into a full-fledged SaaS that opens a new frontier in web development - UI feature flags!.

Problem statement: UI by feature flags

Conditional rendering of web components in React is one thing, prototyping UI's and seeing results directly with the conditional rendering support is another.

Building prototypes and web user interfaces in code editor and saving to see the results in a browser is hard enough without having to apply conditional rendering.

A better approach of using the browser's devtools to directly edit nodes, styles and write scripts is stunted by having no easy save feature and lack supporting conditional rendering without much.

Enter SwitchFlow - a web-based rapid prototyping tool that adds the following to the wheel rather than reinvent it

Building UI with feature flags from existing DevCycle account via their API or created, used and stored locally in the service.
Quick preview of changes with efficiency and speed
Save and resume without user account
Pasting complete HTML markup directly in to the DOM
Importing existing HTML markup to bootstrap the design process
Basic DOM operations - cloning en masse, mutation, traversal, DOM mutation methods and node storage

It is more oriented towards devs but regular users can use its UI feature flagging service.

Demo

Link

https://switch-flow.vercel.app/

Screenshots

Import and export for the web app

Dynamic layout grid for 8px-and-above based designs

The initial screen for the UI feature flags

Providing API keys to use to obtain and use auth token

The variables and features loaded via the API

The UI after selecting an environment in a project

After selecting an element to be flagged

After exporting the page and applying UI flags for previews

As can be seen when comparing the image above to the ones above it, some sections of its UI have been toggled off.

My Code

ogbotemi-2000 / switch-flow

A rapid prototyping design space with support for a novel feature of UI feature flags

SwitchFlow

A web-based rapid prototyping tool that capitalizes on slapping markups from different sources together, editing and saving them. It is intended to make rapid editing and mutation of nodes possible It sports an excellent idea of directly implementing UI feature flags on the UI prototypes to toggle parts of it on or off.

While it is more developer friendly, regular users can using its UI feature flagging functionality with relative ease.

Feature flags on the UI

It supports connecting a DevCycle account using its client and client secret keys through the API and retrieving all variables defined in each

Feature: along with its variations if existent
Project: along with each of its environments - staging, development, production

The app was built around the DevCycle API for the DevCycle Feature Flag Challenge challenge For this reason; custom, local flagging support was not prioritized but will be implemented after the challenge…

View on GitHub

My DevCycle Experience

Delayed due to missing vital info in API docs.

I got delayed a lot by not quickly realizing how to communicate with the DevCycle API because an important information that connects user account authentication with API operations was missing..

There should be some documentation on what to do with the JSON response (I found out after much stress) returned by the curl command below.
I added a code snippet I use further below in JavaScript to interact with the API after receiving the auth JSON response

curl --request POST \
  --url "https://auth.devcycle.com/oauth/token" \
  --header 'content-type: application/x-www-form-urlencoded' \
  --data grant_type=client_credentials \
  --data audience=https://api.devcycle.com/ \
  --data client_id=<client id> \
  --data client_secret=<client secret>

The function I created for obtaining API authentication with consideration for API and network errors

/** This function is defined as async because async functions wrap their return values as promises whose .catch is never called
   * since it wasn't rejected in the first place requiring a boolean check in .then to act as .catch instead
   */

  window.getAuthToken = async function getAuthToken(DEVCYCLE_API_CLIENT_ID, DEVCYCLE_API_CLIENT_SECRET) {
    let error;
    if(getAuthToken.expiresAt > Date.now()) return [getAuthToken.__token, error];
/*window.providerConfig references the form on the page*/
    if(!(DEVCYCLE_API_CLIENT_ID = window.providerConfig.id.value)||!(DEVCYCLE_API_CLIENT_SECRET = window.providerConfig.secret.value)) return;

    return fetch("https://auth.devcycle.com/oauth/token", {
      method: "POST",
      headers: {
        "Content-Type": "application/x-www-form-urlencoded",
      },
      body: new URLSearchParams({
        grant_type: "client_credentials",
        audience: "https://api.devcycle.com/",
        client_id: DEVCYCLE_API_CLIENT_ID,
        client_secret: DEVCYCLE_API_CLIENT_SECRET,
      }),
    }).then(res=>(res.ok||(error=`Request failed due to a HTTP status of ${res.status} for \`https://auth.devcycle.com/oauth/token\`. `), res.json()))
    .then(json=>{
      json.error&&(error &&=error+ `${(json.error_description||'')} • ${json.error}`),
      getAuthToken.auth = json, getAuthToken.expiresAt = Date.now() + json.expires_in * 1000;
      return [getAuthToken.__token = json.access_token, error]
    }).catch(err=>{
      return [null, 'Unable to make requests to `https://auth.devcycle.com/oauth/token` - you are disconnected from the internet • '+ err.message]
    })
  }

It is used as follows and this is the part actually left out of the documentation

fetch(`https://api.devcycle.com/v1/`, {
    method: args.method,
    headers: {
      "authorization": getAuthToken.__token/** references __token for when getAuthToken gets initialized */,
      "content-type": "application/json"
    },
    ...(args.body&&{body: JSON.stringify(args.body)})
    // body: JSON.stringify({ 'description': "A learning experience on DevCycle's API" })
  }).then(res=>res.json());

The API design is top-notch

Aside the hassle above, I really liked how direct the API is where by some requests only use the HTTP verb they contain to retrieve or set information without bloated request bodies.
Whoever crafted the API knew their craft. Kudos!

The SDK Docs

The docs is missing how to handle JSON variables. I found out by guesswork that the default value needs to be parsed into an object for proper behaviour.

Also, the docs should mention that when targeting is turned off for either development, staging, production environments, the devCycleClient.variableValue call will evaluate the default value passed to it regardless of whether the variable is defined as true or not.
This stumped me for quite a while

Additional Prize Categories

API All-Star: I made great use of the API for retrieving SDK keys for all environments, features and variations, variable names

Streaming voice to SQL with AssemblyAI: Execute the generated SQL, use Ollama, RAG templates and vector embeddings

Ogbotemi Ogungbamila — Mon, 25 Nov 2024 07:54:59 +0000

This is a submission for the AssemblyAI Challenge : Sophisticated Speech-to-Text.

What I Built

I built an online Voice to SQL environment for convert the recorded speech of users into SQL statements with the following features:

Voice to SQL

Convert user speech to text, preferably SQL.
Optional feature of streaming the currently recorded voice to the server to display the equivalent SQL statement
Applies intelligence by replacing words in the converted SQL statements with the glyph they are defined as i.e 'less than' gets replaced with '<', in a customizable and extensible widget.
Audio visualizer during record with options to pause and play
Users can specify the bitrate for geeks for optimum results

SQL statements execution
Provides an interface for switching between MySQL and PostgreSQL databases on the fly
Displays details of errors for every database interaction gone wrong

Generation of Vector embeddings, using a RAG widget and PostgreSQL databases: Timescale, Neon.tech

Provides a widget for obtaining embeddings for custom prompts or text, messages from Ollama models running locally
Provides SQL templates: SELECT and INSERT for applying generated embeddings along with their metadata on PostgreSQL databases that support them

Downloads

{query, result} object from executed queries
Recorded audio.
Option to upload {query, result} object to Pinata

Demo

Node.js server on Vercel

https://voice-sql-ai.vercel.app/

Python server for POST requests

https://voice-ai-sql-python.vercel.app/

https://voice-ai-sql-python.vercel.app/upload with {recording: <base64data>} in the POST request body

Psst: GET requests to the Python server still serves the page I copied from https://developer.mozilla.org/en-US/docs/Web/API/MediaStream_Recording_API/Using_the_MediaStream_Recording_API. It was a great, simple demo which I used to learn how to handle base64 encoded and binary data in Python as well as to POST it to AssemblyAI's API.

Screenshots

Enabled dark mode via browser devtools

Expanded view of widget for Voice-to-SQL

View of the other widgets for creating and using vector embeddings

Journey

Falling back to Python

Curiously enough, python code examples AssemblyAI's docs worked while the JavaScript ones in Node.js either crashed with "Not allowed" errors or returned {error: null} as a response via Node.js SDK and API respectively

AssemblyAI's Speech-to-Text API

The API was very straight forward and more flexible than the Python SDK for my use case with the following workflow

Upload binary data from decoded base64 string to AssemblyAI to obtain a URL
Use the received URL along with my API key to request for audio transcription to text and receive the sent JSON.

Usage

I used AssemblyAI's Speech-to-Text to convert recorded speech of users to SQL statements which are then refined further as follows:

Words in the received text are replaced with the glyphs they represent in SQL.

This submission doesn't quite qualify for the additional prompts since I didn't use them but I did something similar to the other two in the webapp I created.

Issues that thwarted the work

Credits issue with real-time and LeMuR

I was not allowed to use the other tools - LeMUR and real-time streaming with the free credits: I was advised to buy credits despite having over $40 worth of free credits, hence why I sort of implemented something similar to them along with speech-to-text on this web app: https://voice-sql-ai.vercel.app/

Real-time streaming

I was going to implement voice to SQL as a stream but the said credits issue got in the way and I got creative by implementing it instead in Speech-to-Text via code.

Final Thoughts

This was a fun project that broadened my knowledge on using python as a server along with Node.js. It also made me add more functionalities to the SQL playground I had built.
Finally, it made me explore how to get creative with handling and sending binary media data in browsers.

Thank you for reading!

DEV Community: Ogbotemi Ogungbamila

ELTAI: web-based alternative to Apache Airflow for MLOps and AI

What I Built

Overview

Demo

How I Used Redis 8

Interactive database session

Custom AI-powered operations

User information storage

AI-pipe: Pipeline for generating/storing embeddings from AI models to DB with data scraped from sites using custom scripts

What I Built

Features

Custom scriptinig

Embeddings generation

Demo

ogbotemi-2000 / ai-pipe

A webapp that scrapes data you tell it to from the internet and lets you cleanse and format it which is then fed to an AI model to generate embeddings

ai-pipe

AI model providers

Ollama

Open AI

PostgresML

Scraping data

Embedding

In Addition

How I Used Bright Data

Scraping browser

Web Unlocker

Qualified Prompts

AI pipeline

Thanks for reading

SwitchFlow: use your DevCycle account variables as UI toggles in prototyping. Export or download the results

What I Built

Problem statement: UI by feature flags

Demo

Link

Screenshots

Import and export for the web app

Dynamic layout grid for 8px-and-above based designs

The initial screen for the UI feature flags

Providing API keys to use to obtain and use auth token

The variables and features loaded via the API

The UI after selecting an environment in a project

After selecting an element to be flagged

After exporting the page and applying UI flags for previews

My Code

ogbotemi-2000 / switch-flow

A rapid prototyping design space with support for a novel feature of UI feature flags

SwitchFlow

Feature flags on the UI

My DevCycle Experience

Delayed due to missing vital info in API docs.

The API design is top-notch

The SDK Docs

Additional Prize Categories

Streaming voice to SQL with AssemblyAI: Execute the generated SQL, use Ollama, RAG templates and vector embeddings

What I Built

Voice to SQL

SQL statements execution

Generation of Vector embeddings, using a RAG widget and PostgreSQL databases: Timescale, Neon.tech

Downloads

Demo

Node.js server on Vercel

Python server for POST requests

Journey

Falling back to Python

AssemblyAI's Speech-to-Text API

Usage

Issues that thwarted the work

Credits issue with real-time and LeMuR

Real-time streaming

Final Thoughts