DEV Community: Amit Wani

Xpenser - AI Expense Manger using Twilio & Gemini AI with WhatsApp Integration

Amit Wani — Mon, 24 Jun 2024 05:31:19 +0000

This is a submission for the Twilio Challenge

What I Built

Xpenser is not just a simple expense manager. It is an advanced expense management tool designed to simplify logging and analyzing your spending habits. With robust features, including seamless WhatsApp integration powered by Twilio, Xpenser stands out as a unique solution for personal finance management.

Demo

Checkout the Demo on YouTube at:

Experience Xpenser live https://xpenser.amitwani.dev.

Explore the project on GitHub:

mtwn105 / xpenser-whatsapp-bot

AI Expense Manger with Twilio & Gemini AI

Xpenser

AI Expense Manager with WhatsApp

Built using Twilio & Gemini AI

🥳 Winner of Twilio x Dev.To Hackathon - https://dev.to/devteam/congrats-to-the-twilio-challenge-winners-2b1n

Demo

You can find Xpenser live here: https://xpenser.amitwani.dev.

Join the Twilio Sandbox WhatsApp Number: +14155238886 by sending join per-mud

Tech Stack

Backend: NodeJS, Express

Frontend: Angular, Typescript

Database: MongoDB

Deployment: Docker, Northflank

AI: Gemini 1.5 Flash

Twilio: Verify, Functions, WhatsApp

Architecture

WhatsApp Integration

Portal

View on GitHub

To test the WhatsApp integration, join the Twilio Sandbox WhatsApp Number: +14155238886 by sending join per-mud

Workflow

User Registration

Users register on the web portal, verifying their phone numbers via OTP (Powered by Twilio Verify).

Dashboard

The dashboard offers detailed analytics, including AI-driven insights, category-wise distribution, and average spending.

Expenses

Users can easily add, manage, and delete expenses. The AI automatically categorizes expenses upon saving.

WhatsApp Integration

Join the Twilio Sandbox WhatsApp Number: +14155238886 by sending join per-mud

Log new expenses effortlessly via WhatsApp. The AI interprets user messages to gather details for saving the expense.

Request insights on spending, such as top categories or average spending. The AI provides answers based on the last 100 expenses.

Tech Stack

Backend: NodeJS, Express
Frontend: Angular, Typescript
Database: MongoDB
Deployment: Docker, Northflank
AI: Gemini 1.5 Flash
Twilio: Verify, Functions, WhatsApp

Architecture

WhatsApp Integration

Portal

Twilio and AI

Twilio

Xpenser leverages three Twilio services: Verify, Functions and WhatsApp Messaging

Verify

Twilio's Verify service sends OTPs for user authentication.

Send OTP

const verification = await client.verify.v2.services(otpServiceId)
      .verifications
      .create({ to: req.body.phonenumber, channel: 'sms' });

Verify OTP

 const verification = await client.verify.v2.services(otpServiceId)
      .verificationChecks
      .create({ to: req.body.phonenumber, code: req.body.code })

Functions

Twilio Functions handles incoming WhatsApp messages, interfacing with the backend and Gemini AI.

Generated functions using

 twilio serverless:init twilio --typescript

export const handler: ServerlessFunctionSignature = async function (
  context: Context<MyContext>,
  event: MyEvent,
  callback: ServerlessCallback
) {
  // Handle Incoming Message 
}

WhatsApp Messaging

Twilio's Programmable WhatsApp Messaging facilitates user interaction through WhatsApp.

Send WhatsApp Message using TwiML

const twiml = new Twilio.twiml.MessagingResponse();
twiml.message("Please ask about spends you have done or spend you want to log.");
callback(null, twiml);

AI

Gemini 1.5 Flash, a Google AI Model, enhances Xpenser with advanced analytics and predictions.

Expense Analytics

Analyzing user expense history to provide insights and respond to user queries.

const analyticsModel = genAI.getGenerativeModel({
    model: "gemini-1.5-flash",
    systemInstruction: "Based on below expenses done by user give answer to user. Optimize as WhatsApp message reply and add emojis.\n\nExpense History:\n" + expenseCsvText,
  });

const generationConfig = {
  temperature: 0.6,
  topP: 0.95,
  topK: 64,
  maxOutputTokens: 8192,
  responseMimeType: "application/json",
};

const chatSession = model.startChat({
  generationConfig
});

const result = await chatSession.sendMessage(event.Body);
const aiResponse = result.response.text()

Expense Categorization

Categorizing expenses based on user-provided descriptions.

const analyticsModel = genAI.getGenerativeModel({
    model: "gemini-1.5-flash",
    systemInstruction: "detect the category and subcategory from below\n1. Housing - Rent, Taxes, Insurance, Utilities, Repairs, Improvement, Fees\n2. Transportation - Payments, Fuel, Insurance, Repairs, Public, Parking, Tolls, Licensing\n3. Food - Groceries, Dining, Coffee, Delivery, Snacks, etc.",
  });

const generationConfig = {
  temperature: 0.6,
  topP: 0.95,
  topK: 64,
  maxOutputTokens: 8192,
  responseMimeType: "application/json",
};

const chatSession = model.startChat({
  generationConfig
});

const result = await chatSession.sendMessage(event.Body);
const aiResponse = result.response.text()

Additional Prize Categories

Xpenser qualifies for the Twilio Times Two prize category by utilizing multiple Twilio features: Verify, Functions, and WhatsApp Messaging.

Future Scope

Spending Habits: Use AI to identify spending patterns and suggest ways to optimize expenses based on historical data.
Mobile Applications: Develop native iOS and Android applications for enhanced accessibility and a more seamless user experience.
Custom Reports: Allow users to generate customizable reports based on various parameters such as date range, categories, or tags.
Investment Tracking: Add features to track investments and provide insights into portfolio performance.
Manage Categories: Create and manage custom categories and sub-categories for a more personalized experience.

Conclusion

It was a great experience to work on this project while exploring Twilio services and integrating AI to enhance the user experience. Thanks to Dev.to team for this amazing hackathon.

Recruitify - Where Talent Meets Opportunity

Amit Wani — Mon, 27 May 2024 05:33:59 +0000

This is a submission for the The AWS Amplify Fullstack TypeScript Challenge

What I Built

Recruitify is a platform designed to bridge the gap between job seekers and employers. It provides a seamless experience for both parties to find the perfect match for their needs.

Features

Employer

Create Job Postings: Employers can create detailed job listings specifying required skills, experience, location, and more.
Manage Applications: Employers can efficiently manage and validate applications received for their job postings.

Job Seeker

Browse Jobs: Job seekers can search and filter job listings based on their preferences.
Apply for Jobs: Easily apply for various job opportunities.
Receive Notifications: Get notified about new job listings that match your skills

Demo and Code

Recruitify is available here and the code is available on GitHub.

mtwn105 / recruitify

Recruitify - Where Talent Meets Opportunity

Introduction

Recruitify is a platform designed to bridge the gap between job seekers and employers. It provides a seamless experience for both parties to find the perfect match for their needs.

Features

Employer

Create Job Postings: Employers can create detailed job listings specifying required skills, experience, location, and more.
Manage Applications: Employers can efficiently manage and validate applications received for their job postings.

Job Seeker

Browse Jobs: Job seekers can search and filter job listings based on their preferences.
Apply for Jobs: Easily apply for various job opportunities.
Receive Notifications: Get notified about new job listings that match your skills

Recruitify is available here

Tech Stack

Backend: AWS Amplify
Frontend: Angular 17, Angular Material, Tailwind CSS
AWS Amplify Technologies: Data (DynamoDB), Authentication (Cognito), Serverless Functions (Lambda), File Storage (S3)
Hosting: AWS Amplify

Database Design

View on GitHub

Screenshots

Browse Jobs Page

Edit Profile

Create/Edit Job Posting

Generate Job Description

Job Details Page (Job Creator)

Job Details Page (Job Seeker)

Job Applications Page (Job Seeker)

Job Applications Page (Job Creator)

New Job Notification

Integrations

Tech Stack

Backend: AWS Amplify
Frontend: Angular 17, Angular Material, Tailwind CSS
AWS Technologies: Data (DynamoDB), Authentication (Cognito), Serverless Functions (Lambda), File Storage (S3), AI (Bedrock)
Hosting: AWS Amplify

Connected Components and Feature Full

This project uses Amplify connected components for Angular - Authenticator for users to sign up and log in.

This project also integrates all four integrations - Data, Authentication, Serverless Functions, File Storage

AWS Amplify

Architecture

Authentication

Utilizes AWS Amplify Authentication for user sign-up and login via email which uses AWS Cognito services.

Configured in amplify/auth/resource.ts

export const auth = defineAuth({
  loginWith: {
    email: true,
  }
});

Implemented with the Amplify UI component <amplify-authenticator> for Angular.

Data

Utilizes AWS Amplify Data to store information about Users, Companies, and Jobs which uses AWS DynamoDB. Database Design is shown above.

Configured in amplify/data/resource.ts

Database Design

Storage

AWS Amplify S3 Storage is used for storing company logos and job seeker resumes.

Configured in amplify/storage/resource.ts

export const storage = defineStorage({
  name: 'recruitifyFiles',
  access: (allow) => ({
    'companyLogos/*': [
      allow.authenticated.to(['read', 'write']),
      allow.guest.to(['read', 'write'])
    ],
    'resumes/*': [
      allow.authenticated.to(['read', 'write']),
      allow.guest.to(['read', 'write'])
    ],
  })
});

The files were uploaded using uploadData like below

 const result = await uploadData({
        data: this.previewUrl,
        path: `companyLogos/${this.authService?.userProfile?.id}`,
      }).result;

And the files were downloaded when necessary using downloadData like below

 downloadData({ path: profile.data[0].resume })

Functions

AWS Amplify Lambda Functions are used to send New Job Notifications to job seekers matching their skill set and Generate Job Description to generate job descriptions for job creators.

Used AWS Amplify AppSync GraphQL API inside the function to find relevant job seekers and create new job notifications.

Used AWS Bedrock with Mistral AI Model inside the function to generate job description.

Configured in amplify/functions/new-job/handler.ts

Triggered when a new job is created by an employer using a mutation in data like the below:

sendJobNotification: a.mutation()
    .arguments({ jobId: a.string(), companyId: a.string() })
    .returns(a.string())
    .authorization(allow => [allow.publicApiKey()])
    .handler(a.handler.function(newJob))

client.mutations.sendJobNotification({
        jobId: job.data?.id,
        companyId: job.data?.companyId
      }).then((job) => {
        console.log(job)
      })

Triggered generate job description like the below:

generateJobDescription: a.mutation()
    .arguments({ title: a.string(), minExperience: a.float(), skills: a.string(), domain: a.string() })
    .returns(a.string())
    .authorization(allow => [allow.publicApiKey()])
    .handler(a.handler.function(generateJobDescription))

client.mutations.generateJobDescription({
      title: this.createJobForm.value.title,
      skills: this.skills.join(','),
      domain: this.createJobForm.value.domain,
      minExperience: this.createJobForm.value.minExperience,
    }).then((response) => {
      console.log(response);
      if (response?.data) {
        this.createJobForm.patchValue({
          description: response.data
        })
      }
    }).finally(() => {
      this.loadingService.hide();
    })

Used Mistral 7B model with prompt like this:

 const text = `
          Job Title: ${title}
          Job Domain: ${domain}
          Job Min Experience: ${minExperience} years
          Job Required Skills: ${skills}
        `
    const prompt = `Your task is to generate a job description for below-given job details.

        Job Details: ${text}

        Job Description: `

And invoke the model like below:

const input = {
      modelId: "mistral.mistral-7b-instruct-v0:2",
      contentType: "application/json",
      accept: "application/json",
      body: JSON.stringify({
        prompt: `<s>[INST] ${prompt} [/INST]`,
        max_tokens: 2000,
        temperature: 0.5,
      }),
    } as InvokeModelCommandInput;

Hosting

Utilised AWS Hosting for seamless deployment of the application.

Future Scope

Application Notifications: Notifications for Job Application status changes for both job seekers and employers
Realtime chat messaging: Realtime chat messaging between job seeker and employer

Conclusion

Recruitify leverages the power of AWS Amplify to deliver a robust, full-stack application. A lot to learn while working with AWS and AWS Amplify toolkit along with Angular on this project.