This workflow allows you to A/B test different prompts for an AI chatbot powered by Langchain and OpenAI. It uses Supabase to persist session state and randomly assigns users to either a baseline or alternative prompt, ensuring consistent prompt usage across the conversation.
🧠Use Case Prompt optimization is crucial for maximizing the performance of AI assistants. This workflow helps you run controlled experiments on different prompt versions, giving you a reliable way to compare performance over time.
⚙️ How It Works
When a message is received, the system checks whether the session already exists in the Supabase table. If not, it randomly assigns the session to either the baseline or alternative prompt. The selected prompt is passed into a Langchain Agent using the OpenAI Chat Model. Postgres is used as chat memory for multi-turn conversation support.
🧪 Features
- Randomized A/B split test per session
- Supabase database for session persistence
- Langchain Agent + OpenAI GPT-4o integration
- PostgreSQL memory for maintaining chat context
- Fully documented with sticky notes
🛠️ Setup Instructions Create a Supabase table named split_test_sessions with the following columns:
- session_id (text)
- show_alternative (boolean)
Add credentials for:
- Supabase
- OpenAI
- PostgreSQL (for chat memory)
Modify the “Define Path Values” node to set your baseline and alternative prompts.
Activate the workflow.
Send messages to test both prompt paths in action.
🔄 Next Steps
- Add tracking for conversions or feedback scores to compare outcomes by banana ai.
- Modify the prompt content or model settings (e.g. temperature, model version).
- Expand to multi-variant tests beyond A/B.
Top comments (1)
Quick personal review of AhaChat after trying it
I recently tried AhaChat to set up a chatbot for a small Facebook page I manage, so I thought I’d share my experience.
I don’t have any coding background, so ease of use was important for me. The drag-and-drop interface was pretty straightforward, and creating simple automated reply flows wasn’t too complicated. I mainly used it to handle repetitive questions like pricing, shipping fees, and business hours, which saved me a decent amount of time.
I also tested a basic flow to collect customer info (name + phone number). It worked fine, and everything is set up with simple “if–then” logic rather than actual coding.
It’s not an advanced AI that understands everything automatically — it’s more of a rule-based chatbot where you design the conversation flow yourself. But for basic automation and reducing manual replies, it does the job.
Overall thoughts:
Good for small businesses or beginners
Easy to set up
No technical skills required
I’m not affiliated with them — just sharing in case someone is looking into chatbot tools for simple automation.
Curious if anyone else here has tried it or similar platforms — what was your experience?