DEV Community: Oyemade Oyemaja

Getting started with Google's Multi-modal "Gemini Pro Vision" LLM with Javascript for Beginners

Oyemade Oyemaja — Fri, 29 Dec 2023 16:02:17 +0000

Building AI-powered Applications w/ Javascript using Langchain JS for Beginners

Getting started w/ Google's Gemini Pro LLM using Langchain JS

In the previous blog posts ⬆️⬆️⬆️, we've discussed building AI-powered apps with different LLMs like Open AI's GPT and Google's Gemini Pro. One thing they all have in common is that they are text-based models which means that they are designed to understand, interpret, and generate text.

How do we then create AI-powered apps that need to interact with images? Well this is where multi-modal LLMs come in, they are LLMs that can understand, interpret, and generate not just text, but also other modes of data such as images, audio, and video.

Google also released a multimodal version of their new Gemini Pro model called Gemini Pro Vison that enables the LLM to interact with images.

In this blog post, we will be using the newly released Gemini Pro Vision LLM to create a nutrition-fact explainer app that takes the image of the nutrition fact on an edible product and returns some insights on it.

Some requirements before we can begin, you need node.js installed on your machine if you do not have it installed you can download it here, we would also need an API key for the Gemini LLM. Here is a guide for creating an API key for Gemini.

To get started:

Navigate to your preferred folder of choice, open it in your code editor, and run npm init -y in the terminal. A package.json file would be generated in that folder.
Edit the package.json file by adding a type property with a value module to the JSON object. "type": "module",.
This indicates that the files in the project should be treated as ECMAScript modules (ESM) by default.
Run npm i @google/generative-ai dotenv in the terminal to install the required dependencies.
Create two (2) new files in the current folder, one called index.js which would house all our code, and another called .env to store our API key. We can also copy the image files to be analysed into the folder.
In our .env file, we need to add our API key. You can do this by typing GOOGLE_API_KEY=YOUR_GOOGLE_API_KEY_GOES_HERE exactly as shown, replacing YOUR_GOOGLE_API_KEY_GOES_HERE with your actual Gemini API key.
In our index.js file we would import the following dependencies:

// This imports the new Gemini LLM
import { GoogleGenerativeAI } from "@google/generative-ai";

// This imports fs from node.js, which we 
// will use to read the image file
import * as fs from "fs";

// This helps connect to our .env file
import * as dotenv from "dotenv";
dotenv.config();

const template = `Task: Analyze images of nutritional facts from various products, which feature diverse designs and include detailed nutritional information typically found on food packaging.

Step 1: Image Recognition and Verification

Confirm if each image is a nutritional facts label.
If the image is not a nutritional facts label, return: "Image not recognized as a nutritional facts label."
Step 2: Data Extraction and Nutritional Analysis

Extract and list the key nutritional contents from the label, such as calories, fats, carbohydrates, proteins, vitamins, minerals and other nutritional details.
Step 3: Reporting in a Strict Two-Sentence Format

First Sentence (Nutritional Content Listing): Clearly list the main nutritional contents. Example: "This product contains 200 calories, 10g of fat, 5g of sugar, and 10g of protein per serving."
Second Sentence (Diet-Related Insight): Provide a brief, diet-related insight based on the nutritional content. Example: "Its high protein and low sugar content make it a suitable option for muscle-building diets and those managing blood sugar levels."
Note: The response must strictly adhere to the two-sentence format, focusing first on listing the nutritional contents, followed by a concise dietary insight.
`;

// Above we created a template variable that 
// contains our detailed instructions for the LLM

// Below we create a new instance of the GoogleGenerativeAI
// class and pass in our API key as an argument
const genAI = new GoogleGenerativeAI(process.env.GOOGLE_API_KEY);

// This function takes a path to a file and a mime type
// and returns a generative part (specific object structure) that 
// can be passed to the generative model
function fileToGenerativePart(path, mimeType) {
  return {
    inlineData: {
      data: Buffer.from(fs.readFileSync(path)).toString("base64"),
      mimeType,
    },
  };
}

// We create the model variable by calling the getGenerativeModel method
// on our genAI instance and passing in an object with the model name
// as an argument. Since we want to interact with images, we use the
// gemini-pro-vision model.
const model = genAI.getGenerativeModel({ model: "gemini-pro-vision" });

// We create our image variable by calling the fileToGenerativePart function
// and passing in the path to our image file and the mime type of the image.
const image = [fileToGenerativePart("./pepsi.jpeg", "image/jpeg")];

// We call the generateContent method on our model and pass in our template
// and image variables. This returns a promise, which we await and assign
// to the result variable. We then await the response property of the result
// variable and assign it to the response variable. Finally, we await the text
// method of the response variable and assign it to the text variable.
const result = await model.generateContent([template, ...image]);
const response = await result.response;
const text = response.text();

// Log the text variable to the console.
console.log(text);

Now we would run node index.js in our terminal. We should then see a result similar to this in our console:
This product contains 150 calories, 0g of fat, 42g of carbohydrates, 41g of sugar, and 0g of protein per 355ml serving. Given the high sugar content, moderate consumption is advisable for those watching their sugar intake.
This is the image that was analyzed

Keep in mind that each time you run the model, the results might vary slightly. However, the overall meaning and context should remain consistent.

Here is a link to the GitHub repo

Don't hesitate to ask any questions you have in the comments. I'll make sure to respond as quickly as possible.

Getting started w/ Google's Gemini Pro LLM using Langchain JS

Oyemade Oyemaja — Sun, 17 Dec 2023 02:10:24 +0000

Building AI-powered Applications w/ Javascript using Langchain JS for Beginners
In the previous blog post ⬆️⬆️⬆️, we talked about getting started with AI-powered apps with Langchain JS, we then went on to build an AI-powered emoji explainer app that uses the OpenAI LLM to power it.

In this blog post, we will be using the newly released Gemini Pro LLM to create the emoji explainer app. More details about the Gemini AI Model here.

To get started:

Navigate to your preferred folder of choice, open it in your code editor, and run npm init -y in the terminal. A package.json file would be generated in that folder.
Edit the package.json file by adding a type property with a value module to the JSON object. "type": "module",.
This indicates that the files in the project should be treated as ECMAScript modules (ESM) by default.
Run npm i langchain @langchain/google-genai dotenv in the terminal to install the required dependencies.
Create two (2) new files in the current folder, one called index.js which would house all our code, and another called .env to store our API key.
In our .env file, we need to add our API key. You can do this by typing GOOGLE_API_KEY=YOUR_GOOGLE_API_KEY_GOES_HERE exactly as shown, replacing YOUR_GOOGLE_API_KEY_GOES_HERE with your actual Gemini API key.
In our index.js file we would import the following dependencies:

// This imports the new Gemini LLM
import { ChatGoogleGenerativeAI } from "@langchain/google-genai";

// This imports the mechanism that helps create the messages
// called `prompts` we send to the LLM
import { PromptTemplate } from "langchain/prompts";

// This imports the tool called `chains` that helps combine 
// the model and prompts so we can communicate with the LLM
import { LLMChain } from "langchain/chains";

// This helps connect to our .env file
import * as dotenv from "dotenv";
dotenv.config();

const template = `Interpret the emotions conveyed by the following emoji(s): {emojis}. Consider each emoji individually and in combination with others in the series. Apply your knowledge of common emoji usage, cultural contexts, and emotional expressions to provide an insightful interpretation. When analyzing multiple emojis, consider the sequence and combination of the emojis to understand any nuanced emotional narrative or sentiment they may collectively express. Additionally, take into account the following considerations:

Emoji Specificity: Identify each emoji and any specific emotions or ideas it typically represents.
Cultural Context: Acknowledge if certain emojis have unique meanings in specific cultural contexts.
Emotional Range: Recognize if emojis express a range of emotions, from positive to negative or neutral.
Sequential Interpretation: When multiple emojis are used, analyze if the sequence changes the overall emotional message.
Complementary and Contrasting Emotions: Note if emojis complement or contrast each other emotionally.
Common Usage Patterns: Reflect on how these emojis are commonly used in digital communication to infer the underlying emotions.
Sarcasm or Irony Detection: Be aware of the possibility of sarcasm or irony in the use of certain emojis.
Emoji Evolution: Consider any recent changes in how these emojis are used or perceived in digital communication.

Based on these guidelines, in one very short sentence, provide a short interpretation of the emotions conveyed by the inputted emoji(s). `;

const promptTemplate = new PromptTemplate({
  template,
  inputVariables: ["emojis"],
});

// Above we created a template variable that contains our
// detailed instructions for the LLM, we also added a 
// variable {emojis} which would be replaced with the emojis
// passed in at runtime.
// We then create a prompt template from the template and
// input variable.


// We create our model and pass it our model name 
// which is `gemini-pro`. Another option is to pass
// `gemini-pro-vision` if we were also sending an image
// in our prompt
const geminiModel = new ChatGoogleGenerativeAI({
  modelName: "gemini-pro",
});

// We then use a chain to combine our LLM with our 
// prompt template
const llmChain = new LLMChain({
  llm: geminiModel,
  prompt: promptTemplate,
});

// We then call the chain to communicate with the LLM
// and pass in the emojis we want to be explained.
// Note that the property name `emojis` below must match the
// variable name in the template earlier created.
const result = await llmChain.call({
  emojis: "😂🤣",
});

// Log result to the console
console.log(result.text);

Now we would run node index.js in our terminal. We should then see a result similar to this in our console: 😂🤣: Laughing out loud with tears of joy, expressing intense amusement or humor.

Keep in mind that each time you run the model, the results might vary slightly. However, the overall meaning and context should remain consistent.

Here is a stackblitz with a demo:

Don't hesitate to ask any questions you have in the comments. I'll make sure to respond as quickly as possible.

Building AI-powered Applications w/ Javascript using Langchain JS for Beginners

Oyemade Oyemaja — Sat, 16 Dec 2023 17:56:05 +0000

AI is here to stay. As developers, it's our responsibility to not only develop applications that help improve the quality of life of the human race but also make sure to do it as quickly as possible by taking advantage of the best technologies around so we can do more with less.

In this tutorial, I will introduce Langchain JS which is a framework that simplifies building AI-powered applications by providing all the components needed to build them in one place.

Langchain works with a myriad of LLMs including OpenAI's GPT, Google's Gemini, HuggingFace, and many more. A complete list can be found here.

Quick start: We will be building an emoji explainer app to help describe the emotions behind an emoji or series of emojis.
For example, if these emojis are inputted 😂🤣 we should return something similar to The emojis 😂🤣 convey a sense of uncontrollable laughter and joy, often used to express extreme amusement or hilarity.

Some requirements before we can begin, you need node.js installed on your machine if you do not have it installed you can download it here, we would also need an API key from the LLM of choice, this case, we would be using OpenAI. Here is a guide on creating an API key for OpenAI.

To get started:

Navigate to your preferred folder of choice, open it in your code editor, and run npm init -y in the terminal. A package.json file would be generated in that folder.
Edit the package.json file by adding a type property with a value module to the JSON object. "type": "module",.
This indicates that the files in the project should be treated as ECMAScript modules (ESM) by default.
Run npm i langchain dotenv in the terminal to install the required dependencies.
Create two (2) new files in the current folder, one called index.js which would house all our code, and another called .env to store our API key.
In our .env file, we need to add our API key. You can do this by typing OPENAI_API_KEY=YOUR_OPEN_AI_KEY_GOES_HERE exactly as shown, replacing YOUR_OPEN_AI_KEY_GOES_HERE with your actual OpenAI key.
In our index.js file we would import the following dependencies:

// This imports the OpenAI LLM
import { OpenAI } from "langchain/llms/openai";

// This imports the mechanism that helps create the messages
// called `prompts` we send to the LLM
import { PromptTemplate } from "langchain/prompts";

// This imports the tool called `chains` that helps combine 
// the model and prompts so we can communicate with the LLM
import { LLMChain } from "langchain/chains";

// This helps connect to our .env file
import * as dotenv from "dotenv";
dotenv.config();

const template = `Interpret the emotions conveyed by the following emoji(s): {emojis}. Consider each emoji individually and in combination with others in the series. Apply your knowledge of common emoji usage, cultural contexts, and emotional expressions to provide an insightful interpretation. When analyzing multiple emojis, consider the sequence and combination of the emojis to understand any nuanced emotional narrative or sentiment they may collectively express. Additionally, take into account the following considerations:

Emoji Specificity: Identify each emoji and any specific emotions or ideas it typically represents.
Cultural Context: Acknowledge if certain emojis have unique meanings in specific cultural contexts.
Emotional Range: Recognize if emojis express a range of emotions, from positive to negative or neutral.
Sequential Interpretation: When multiple emojis are used, analyze if the sequence changes the overall emotional message.
Complementary and Contrasting Emotions: Note if emojis complement or contrast each other emotionally.
Common Usage Patterns: Reflect on how these emojis are commonly used in digital communication to infer the underlying emotions.
Sarcasm or Irony Detection: Be aware of the possibility of sarcasm or irony in the use of certain emojis.
Emoji Evolution: Consider any recent changes in how these emojis are used or perceived in digital communication.

Based on these guidelines, in one very short sentence, provide a short interpretation of the emotions conveyed by the inputted emoji(s). `;

const promptTemplate = new PromptTemplate({
  template,
  inputVariables: ["emojis"],
});

// Above we created a template variable that contains our
// detailed instructions for the LLM, we also added a 
// variable {emojis} which would be replaced with the emojis
// passed in at runtime.
// We then create a prompt template from the template and
// input variable.


// We create our model and pass it a `temperature` of 0.5
// The `temperature` represents how creative we want 
// the model to be and it takes values 0.1 - 1.
// A lower temperature (closer to 0.1) makes the model's
// responses more predictable and conservative. On the 
// other hand, a higher temperature (closer to 1) allows for
// more creativity and variation in responses.
const openAIModel = new OpenAI({
  temperature: 0.5,
});

// We then use a chain to combine our LLM with our 
// prompt template
const llmChain = new LLMChain({
  llm: openAIModel,
  prompt: promptTemplate,
});

// We then call the chain to communicate with the LLM
// and pass in the emojis we want to be explained.
// Note that the property name `emojis` below must match the
// variable name in the template earlier created.
const result = await llmChain.call({
  emojis: "😂🤣",
});

// Log result to the console
console.log(result.text);

Now we would run node index.js in our terminal. We should then see a result similar to this in our console: The emojis 😂🤣 convey a sense of extreme amusement and laughter, possibly indicating that the sender found something very funny.

Keep in mind that each time you run the model, the results might vary slightly. However, the overall meaning and context should remain consistent.

Here is a stackblitz with a demo:

Don't hesitate to ask any questions you have in the comments. I'll make sure to respond as quickly as possible.

Improve performance in an Angular app Using Brotli and Gzip Text Compression

Oyemade Oyemaja — Mon, 25 Apr 2022 12:14:49 +0000

Text compression essentially reduces the file size/payload of text-based resources, decreasing the amount of data transferred. A compressed text-based resource is delivered to your browser faster therefore text-based resources should be served with compression to minimise total network bytes.

Brotli and Gzip are compression algorithms that can significantly improve the overall bundle size of an application by at least 50% thereby improving performance. In this blog post we would look at how we can also generate Brotli and Gzip versions of our bundle so that browsers that support them can request for them or fall back to our non compressed version if they don't support it.

To achieve this in angular we have to extend the builder used when building our application for production to allow us run additional webpack plugins during the build process. The plugins needed are:

"compression-webpack-plugin" for our Gzip files
"brotli-webpack-plugin" for Brotli files

To implement this technique:

Run npm install -D compression-webpack-plugin brotli-webpack-plugin to install the packages as dev dependencies.
Create a custom webpack config (custom-webpack.config.js) file in your src folder that would run our plugins. Paste the following into the file.

const CompressionPlugin = require("compression-webpack-plugin");
const BrotliPlugin = require("brotli-webpack-plugin");

module.exports = {
  plugins: [
    new CompressionPlugin({
      algorithm: "gzip",
    }),
    new BrotliPlugin(),
  ],
};

Run npm i -D @angular-builders/custom-webpack this would install the builder that would allow us to run additional webpack configurations.
In the angular.json file under the build property replace the value of the builder property and add a new property called options with its value as thus:

...

"builder": "@angular-builders/custom-webpack:browser",
"options": {
    "customWebpackConfig": {
        "path": "src/custom-webpack.config.js",
        "replaceDuplicatePlugins": true
    }
}

....

 Once the above has been completed run ng build. If it runs successfully, go into the dist/projectName folder you should find .br and .gz versions of all .js files which should be significantly smaller in size.

Also note that depending on the server in which the application would be deployed some configuration might be needed to support .br files but it's good to know that most servers support them.

BONUS

This technique can applied to all text-based files for the web especially css, with the help of brotli, css files can be compressed up to 70% of its initial size because it has a large number of repeated strings which helps compression. Please note that after compressing these files "Content-Encoding: br" and "Content-Type" headers need to be set on them when deploying them to the server so that the browser understands them.

I created a javascript project that can help compress files with the brotli algorithm using webpack plugins on github. Feel free to make use of it to compress your text-based web files.

Here are some links on how to setup compression on NGINX and APACHE servers:

Reviewed By: Sander Elias🙏🏽

Managing table filters between components

Oyemade Oyemaja — Sun, 16 May 2021 23:45:43 +0000

TLDR; The logic behind this is to pass the table filters as query params when exiting the table route and when trying to re-activate the table route add the {queryParamsHandling="preserve”} directive after the routerLink so that the query params would be preserved.

Introduction
Tables are very important features when it comes to building applications, depending on the type of application, tables can range from small rows and columns of data, to really huge dataset representations in tabular form. Regardless of this, being able to filter through the data on a table is a very important part of the overall user experience.

After implementing filters, another issue that arises is how to identify previously applied filters after a component containing a table has been destroyed. Here is a use case: We have a component showing a list of customers in a table with filters and each row ends with an open button. Let’s assume a user of the application decides to filter the customers by name and he’s left with a particular customer and clicks open. In this scenario, the open button takes the user to a new component that gives more insight into the customer's history than can be represented on a table. In the use case above, all filters applied are lost once the user navigates away from the customer's table component and would be set to default when the component is reinitialized which might be a very bad user experience.

Possible Solutions

Using a state management tool to manage the filter state between the components e.g NgRx, Akita, etc.
Using a BehaviourSubject to store the filter state.
Pass the filter state as query params

Implementation of the solution
We would be applying the third solution because it is very easy to implement and takes less effort.
Steps:

When routing to another page from our table component we want to pass our filter values as query params. This can be done from the template by using this syntax:

or from the component by:
When trying to route back to the table component, the {queryParamsHandling="preserve”} directive must be set along with the routerLink directive so that the query params on the route can be preserved.
In the table component when initializing the filters, they should be set to the value of the query params or an empty string if there is no query param present so that previous filters can be populated. This can be done using this syntax:

With this setup, you have successfully been able to manage your filter state without any extra setup or state management solution.

Thanks for reading.

Reviewed by
Bonnie Brennan #ngMom
Sander Elias