DEV Community: raghab

Recommendation algorithm with vector database

raghab — Mon, 16 Oct 2023 11:09:57 +0000

In this article, we’ll explore how to build a simple movie recommendation system using vector databases and Node.js. We’ll utilize pgVector, a PostgreSQL extension that enables scalable vector computations, to create a vectorised movie dataset and perform nearest neighbor searches to generate recommendations.

First, let’s start by discussing the importance of vector databases and their advantages over traditional relational databases.

Why Vector Databases?

Traditional relational databases are great for storing structured data, but they fall short when it comes to handling large amounts of unstructured or semi-structured data. This is where vector databases come in.

Vector databases are designed to handle high-dimensional data, such as vectors and matrices, efficiently. They offer several benefits over traditional relational databases, including:

Scalability

Vector databases are optimized for handling large datasets and can scale horizontally by adding more nodes to the cluster. This makes them ideal for applications that require processing massive amounts of data.

Speed

Vector databases are designed for speed and can perform calculations much faster than traditional relational databases. They achieve this through the use of specialized hardware, such as GPUs or TPUs, and optimized algorithms.

Flexibility

Vector databases allow for various data types, including numerical, text, and image data. They also support various query languages, such as SQL, Python, and Rust.

A Simple Movie Recommender

First, let’s start by setting up the project and installing the required dependencies:

npm init -y
npm install express pg sequelize pgvector openai-api

Now, let’s create a new Express.js server and connect to our PostgreSQL database using Sequelize:

const express = require('express');
const Sequelize = require('sequelize');
const { Op } = Sequelize;
const app = express();

const sequelize = new Sequelize('movies', 'your_username', 'your_password', {
  host: 'localhost',
  dialect: 'postgres'
});

app.use(express.json());

const Movie = sequelize.define('Movie', {
  id: {
    type: Sequelize.INTEGER,
    primaryKey: true,
    autoIncrement: true
  },
  title: {
    type: Sequelize.STRING
  },
  description: {
    type: Sequelize.TEXT
  },
  genre: {
    type: Sequelize.STRING
  },
  director: {
    type: Sequelize.STRING
  },
  actors: {
    type: Sequelize.ARRAY(Sequelize.STRING)
  },
  releaseYear: {
    type: Sequelize.INTEGER
  }
}, {
  hooks: {}
});

app.get('/movies', async (req, res) => {
  const movies = await Movie.findAll({
    where: {
      genre: { [Op.like]: '%action%' }
    }
  });
  res.json(movies);
});

app.listen(3000, () => {
  console.log('Server listening on port 3000');
});

Next, we’ll add the pgvector plugin to our PostgreSQL database:

CREATE EXTENSION IF NOT EXISTS pgvector;

Now, we can create a vector field for our Movie model using the pgvector module:

const { Vector } = require('pgvector');

const movieVector = new Vector(sequelize, Movie, 'movie_vectors', {
  fields: ['title', 'description', 'genre'],
  dimension: 128
});

Here, we create a new vector field called movieVectors for our Movie model, using the fields option to specify the columns we want to include in the vector (in this case, title, description, and genre). We also set the dimension option to 128, which determines the number of dimensions in the vector space.To generate vectorized embeddings for our movies, we’ll use the OpenAI Embeddings API:

const openai = require('openai-api');

const api = new openai.Api('YOUR_API_KEY');

const movieTitles = ['The Shawshank Redemption', 'The Godfather', 'The Dark Knight'];

async function fetchEmbeddings() {
  const response = await api.encode({
    inputs: movieTitles,
    output_type: 'json',
  });

  const embeddings = response.data.map(( embedding ) => ({
    title: movieTitles[embedding.input_index],
    embedding,
  }));

  console.log(embeddings);
}

fetchEmbeddings();
here’s another direct approach we can use!

// Set up OpenAI client
const openaiClient = new openai.OpenAIClient(process.env.OPENAI_API_KEY);

// Define a function to get recommendations
async function getRecommendations(input) {
  // Get embeddings from OpenAI
  const embeddings = await openaiClient.embedding.get(input);

  // Calculate similarity between input and existing reviews
  const similarities = await calculateSimilarity(pool, embeddings);

  // Return top N recommended reviews
  return similarities.slice(0, 5);
}

// Define a function to calculate similarity between two vectors
async function calculateSimilarity(pool, vec1, vec2) {
  // Calculate cosine similarity between two vectors
  const similarity = Math.abs(vec1.dot(vec2));

  // Normalize similarity to get a score between 0 and 1
  similarity /= (Math.sqrt(vec1.length * vec2.length));

  return similarity;
}

// Endpoint to get recommendations
app.post('/recommendations', async (req, res) => {
  const input = req.body.input;
  const recommendations = await getRecommendations(input);
  res.json(recommendations);
});

In conclusion, implementing a recommendation system using vector databases and the OpenAI Embeddings API can be a powerful way to provide personalized suggestions to users. By leveraging the capabilities of natural language processing and machine learning, this approach allows developers to create intelligent systems that can understand and analyze text data, and make informed decisions based on it.The benefits of this approach are numerous.

Firstly, it enables the creation of recommendation systems that are not limited by the traditional constraints of collaborative filtering or content-based filtering methods. Instead, vector databases can capture the nuances and complexities of human language, allowing for more accurate and diverse recommendations.

Secondly, the use of pre-trained models like the OpenAI Embeddings API can significantly reduce the amount of training data required, making it easier and faster to develop and deploy these systems.

Finally, this approach can also improve the scalability and efficiency of recommendation systems, as vector databases can handle large amounts of data and perform calculations much faster than traditional relational databases.

However, there are also some challenges and limitations to consider when implementing this approach. One of the main challenges is the quality of the embeddings themselves, which can vary depending on the specific model used and the quality of the training data.

Additionally, there may be issues with dimensionality, where the number of features in the vector space becomes too high, leading to overfitting or decreased performance. Careful tuning and evaluation of the model can help mitigate these risks.

Despite these challenges, the potential benefits of using vector databases and the OpenAI Embeddings API for recommendation systems make it an exciting area of research and development. As the capabilities of natural language processing and machine learning continue to evolve, we can expect to see even more innovative applications of this technology in the future. Whether you’re a developer looking to build a cutting-edge recommendation system or a business looking to improve customer engagement, this approach is definitely worth considering.

originally published here at my blog

framework7- build ios, android styled apps with JavaScript

raghab — Mon, 09 Oct 2023 10:42:55 +0000

For a JavaScript developer it is a very tough task to build mobile application these days, not only because handling the UI is an exhaustive task but also because in most of the frameworks, you are quite limited when it comes to applying what you already know.

framework 7 presents itself in a sweet spot where it gives you tools to create your own user interface either with iOS style or with Android material you design it provides excellent UI components, for overall development of your application.

npm i framework7-cli -g
framework7 create --ui

simply navigate ahead and choose your working stack!

if you don’t want to use framework 7 and would settle on less functionality you can also use konsta UI which offers almost the same experience!

Components

Dialog

<script>
  let dialogOpened = false;

  function openDialog() {
    dialogOpened = true;
  }

  function closeDialog() {
    dialogOpened = false;
  }
</script>

<main>
  <button on:click={openDialog}>Open Dialog</button>

  {#if dialogOpened}
    <div class="dialog">
      <div class="dialog-content">
        <h2>This is a Svelte dialog</h2>
        <button on:click={closeDialog}>Close</button>
      </div>
    </div>
  {/if}
</main>

Action Sheet

<script>
  let actionSheetOpened = false;

  function openActionSheet() {
    actionSheetOpened = true;
  }

  function closeActionSheet() {
    actionSheetOpened = false;
  }
</script>

<main>
  <button on:click={openActionSheet}>Open Action Sheet</button>

  {#if actionSheetOpened}
    <div class="action-sheet">
      <div class="action-sheet-buttons">
        <button on:click={closeActionSheet}>Cancel</button>
        <button on:click={closeActionSheet}>Button 1</button>
        <button on:click={closeActionSheet}>Button 2</button>
      </div>
    </div>
  {/if}
</main>

Accordion

<script>
  let accordionOpened = false;

  function toggleAccordion() {
    accordionOpened = !accordionOpened;
  }
</script>

<main>
  <button on:click={toggleAccordion}>Toggle Accordion</button>

  {#if accordionOpened}
    <div class="accordion">
      <div class="accordion-header" on:click={toggleAccordion}>
        <span>Accordion Header</span>
      </div>
      <div class="accordion-content">
        <p>This is the accordion content</p>
      </div>
    </div>
  {/if}
</main>

Building mobile apps can be a daunting task for JavaScript developers. The challenges of handling intricate UI designs and the limitations posed by many frameworks can be overwhelming. However, Framework7 and Konsta UI come to the rescue, offering some excellent solutions.

Framework7 finds the sweet spot by providing tools to craft user interfaces in iOS or Android styles. Its robust components simplify the overall app development journey. The simplicity of getting started, as seen in the provided CLI commands, makes Framework7 a practical choice.

On the other hand, Konsta UI offers a comparable experience, making it a noteworthy alternative. Both frameworks empower developers with essential components like dialogs, action sheets, and accordions, as demonstrated in the code snippets.

The beauty of these options lies in their flexibility, allowing developers to choose based on project requirements and personal preferences. With these frameworks, JavaScript developers can transform the challenging task of mobile app development into a more manageable and creative process, leading to engaging user experiences. Happy coding!

originally published here at my blog

instant distributed database- gun.js

raghab — Sun, 08 Oct 2023 16:19:56 +0000

GUN is like a super cool toolkit for building apps that need to share data securely. It’s all about keeping things decentralized and private, allowing developers to create applications similar to popular services like Dropbox. With GUN, you can work with different types of data, from simple values to complex documents and videos, making it really versatile. Plus, it works offline and ensures that your data stays encrypted and safe. Many developers love it for creating alternatives to mainstream apps, and there’s a great community behind it, making it even better!

it is easy to import in your project

<script src="https://cdn.jsdelivr.net/npm/gun/gun.js"></script>

here we have a basic example using gun to add an entry and listen to it.

const key = 'sample-key';
const data = {
  message: 'Hello, GUN!'
};

// Store data in the GUN database
gun.get(key).put(data, ack => {
  if (ack.err) {
    console.error('Error storing data:', ack.err);
  } else {
    console.log('Data stored successfully!');
  }
});

// Retrieve data from the GUN database in realtime
gun.get(key).on(storedData => {
  if (storedData) {
    console.log('Retrieved data:', storedData);
  } else {
    console.log('No data found for the key:', key);
  }
})

That is all you need to get started with gun!

gunjs provides an incredibly simple interface for programming and eases out most of the headaches, such ss authentication. in the next example you’ll see how easy it is to authenticate with gunjs.

const gun = Gun();
const user = gun.user();

function register() {
    user.create(username, password, (ack) => {
        if (ack.err) {
            console.error(ack.err);
        } else {
            console.log('User created successfully:', ack);
        }
    });
}

function login() {
    user.auth(username, password, (ack) => {
        if (ack.err) {
            console.error(ack.err);
        } else {
            console.log('User logged in successfully:', ack);
        }
    });
}

I don’t think it can get more effortless than that! gun.js has a lot to offer, it is for most the convenient and the fastest tool i have used to create prototypes and scalable applications!

stay tuned for more!