This is a submission for the Cloudflare AI Challenge.
What I Built
This project is a Cloudflare serverless worker with an AI service binding deployed for processing either audio, image or text submitted by users into semantic useful information, especially in the context for submitting expenses.
For users that prefer having a UI to interact with, a simple NextJS web app is created to demonstrate how to upload files and utilizes API routes to make a POST request to the Cloudflare worker endpoint.
How it works
The diagram below looks a bit blurry on this platform, please refer to the raw image here
User can pass any of the following types of content - audio, image or text. Based on the input type given, we will process them eventually to form JSON like this that can be used by any app to display useful expense information for them.
const SAMPLE_RECORD: Expense = {
date: "2024-04-11",
time: "14:00",
type: "product",
item: "Rubber Ducky",
expenditure: 100.0,
};
Models used in this project:
- Image-to-text: @cf/unum/uform-gen2-qwen-500m
- Automatic Speech Recognition: @cf/openai/whisper
- Text Generation: @hf/thebloke/mistral-7b-instruct-v0.1-awq
Demo
Try out the worker!
The worker is deployed at https://cf-journal.senchatea.workers.dev.
Text input
curl --location 'https://cf-journal.senchatea.workers.dev?type=text' \
--header 'Content-Type: text/plain' \
--data '2 weeks ago, I went to eat a buffet at Swensens Unlimited at the T2 airport, it'\''s really nice but it costs like $36 per person after GST, and there were 2 of us.'
Audio input
curl --location 'https://cf-journal.senchatea.workers.dev?type=audio' \
--header 'Content-Type: application/octet-stream' \
--header 'Authorization: Bearer pYrzMvsyURxsCUeaQDsa3lSO_tBDQEuiPB3iLEQt' \
--data '@postman-cloud:///1eef4b5b-a64c-4c10-ad4b-5f22c528880b'
Image input
curl --location 'https://cf-journal.senchatea.workers.dev?type=image' \
--header 'Content-Type: application/octet-stream' \
--header 'Authorization: Bearer pYrzMvsyURxsCUeaQDsa3lSO_tBDQEuiPB3iLEQt' \
--data '@postman-cloud:///1eef4b68-e5d7-49e0-a28d-f60b3bd7d70e'
Try it in the web app!
https://cf-journal.vercel.app/
My Code
The project has 2 folders:
-
serverless: Code that is deployed to a Cloudflare worker. You can find the text generation prompt & code to use the AI models atutils.ts. The Cloudflare worker request handler is found atmain.ts -
app: Nextjs app code that is deployed to Vercel
Introduction
This project is a Cloudflare serverless worker with an AI service binding deployed for processing either audio, image or text submitted by users into semantic useful information, especially in the context for submitting expenses.
For users that prefer a UI, a simple NextJS web app is created to demonstrate how to upload files and utilizes API routes to make a POST request to the Cloudflare worker endpoint.
How it works
Models used
From Cloudflare AI Models:
- Image-to-text: @cf/unum/uform-gen2-qwen-500m
- Automatic Speech Recognition: @cf/openai/whisper
- Text Generation: @hf/thebloke/mistral-7b-instruct-v0.1-awq
Try out the worker!
The worker is deployed at https://cf-journal.senchatea.workers.dev.
Text input
curl --location 'https://cf-journal.senchatea.workers.dev?type=text' \
--header 'Content-Type: text/plain' \
--data '2 weeks ago, I went to eat a buffet at Swensens Unlimited at the T2 airport, it'\''s really nice but it costs like $36 per person after GST, and there were…Journey
How the idea came about
Before OpenAI introduced ChatGPT and revitalized the AI scene, I once created Billy, a cute a little expense tracker app to help my mom to track her expenses easier. It actually received good reception with my friends both online & offline because of the clean and straight forward UI.
However, the user retention wasn't very good because the act of logging expenses is very tedious because of the multitude of fields to fill. It was very difficult to encourage the habit of logging expenses for the users.
For many users, it is much direct and easier to talk or take a picture with a phone as compared to typing. But it was rather difficult to interpret information from the data of these input types back then, so I dropped that project eventually.
Now, by utilizing different AI models, we can interpret and organize important information for them more easily, simplifying the expense submission process.
This project is meant to be a MVP to demonstrate that.
Room for improvements
I'm very new to AI so there will definitely a lot of things to improve on!
The webapp in this submission is a very early MVP of what I envision the actual expense tracker app to be.
- For now, it only create expenses based on the information in the file input (audio/mp3/text).
- Ideally after submission, you can still modify the individual fields of the expense form. The AI portion is just there to assist the auto-filling process.
I'm also not very familiar with prompt engineering, but I would also like to improve the text generation prompt in the future.
Resources
These are some helpful resources from Cloudflare that I have used to work on this project since I wasn't familiar with the AI models available for text generation/inference.
My project is eligible for the Triple Task type category.
Top comments (4)
Great post. I've also been thinking about a full fledged expense tracker but I want it to be able to read and understand bank statements. The problem with bank statement is that it is not very descriptive hence very difficult to categorise the expenses :( Any tips?
Thanks for the qn! Below is my take on it.
For bank statements, usually they have acronyms or codes for specific transactions - (example), so that might narrow the category down a little. Next, they should at least include the name or a short form of the name of the company responsible for the product/activity u bought so that's also good for categorizing. If for some reason, it doesn't have either, then it may be better to corroborate the bank statements with receipts that are clearer in what that thing was for and get users input directly what those refer to. Then train a model for such outlier items.
In my country I get a sms to all the transactions. I’m actually writing one to phrase all those data to a json so I can process it. iPhone shortcut will send sms to a web hook when it receive a sms.
An expense journal integrated with Cloudflare AI offers advanced expense management capabilities. internet for seniors Leveraging Cloudflare's AI-powered analytics, the system can automatically categorize expenses, detect anomalies, and generate insightful reports in real-time. By harnessing the scalability and security of Cloudflare's infrastructure, businesses can streamline expense tracking processes, enhance financial visibility, and make data-driven decisions to optimize budget allocation and control costs effectively.