OctoGrad/DevGrad 2020 - Banter.ly

Amir — Sun, 24 May 2020 03:07:46 +0000

My Final Project

Not so much a school project as a passion project that forced me to leverage almost everything I learned in the past 3 years of University:

I present to you Banter.ly, a little web app that one day aims to be the the world's most comprehensive open source chat analytics 🔎 and visualization app 📊.

The idea for Banter.ly came when I was trying to analyze some of my own chats. All the other chat analyzers I found online were either too shallow (i.e. did not have what I was looking for), or had sketchy privacy statements.

So like and any good dev, since I couldn't find it, I built it.

Demo Link

You can try it out for yourself (sorry about the http, don't want to setup certs or a domain before v0.2)

Or if it's bugging out (which frankly is quite likely 😅), checkout this demo of what an analyzed group chat looks like https://www.loom.com/share/d18297dbc3964fe9ad8f7801a5f386c9

Link to Code

sahaab-io / banter.ly

The world's most comprehensive open source chat analytics 🔎 and visualization app 📊

Banter.ly

Banter.ly's goal is simply to be the world's most comprehensive open source chat analytics 🔎 and visualization app 📊

We want to make it easy and safe for anyone to get the most advanced insights from their text conversations with the click of a button

See a live demo here

Features

Banter.ly currently only supports Whatsapp chat exports - more messengers and formats are on the roadmap (feature requests and cool ideas are welcome!)

Frequency analysis
Sentiment, emotion, and profanity analysis
Named entity extraction

Getting Started

Banter.ly is built on top of Dash, which provides a fantastic python environment for creating dashboards

Prerequisites

Python 3.7.7
MongoDB running locally or one their free hosts

Running the App

# install poetry
curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py | python
# clone this repo and go into it
git clone https://github.com/sahaab-io/banter.ly.git && cd banter.ly
# create and activate the virtualenv + install dependencies
poetry

…

View on GitHub

How I built it

At first, Banter.ly was just a notebook on Google Collab where I was trying to plot some cool interactive visualizations, but it quickly grew out of scope. Plus I realized that others out there could benefit from this so I decided to turn it into a web app, one that you could also run locally if you don't want your super personal conversations to ever leave your PC.

When building Banter.ly, I only used open-source tech (except an S3 bucket), with the main components being MongoDB and a monolith made with Dash (which is built on top of a bunch of other open-source tech, namely Flask and React).

Now Dash is the really interesting piece here. If you ever dreamed of writing JavaScript in Python, then congratulations the devs over at Plot.ly have granted your wish; Dash transforms your python code into React code which then becomes honest-to-god-true-blue browser rendered HTML+CSS+JS. It's also recently passed the 1.0 mark and after using it I'd say it's more than solid enough for production use.

That said, while it's nice to have a monolith repo for handling the data processing and UI, it turns out for this particular use case it would have been better to split the app into an independently scalable front-end and back-end; the two big hurdles in usability were the one-directional flow of doing things that Dash forces on you (for good reason, but not for this use case), plus the amount of time (and cost) it takes to process some of the larger requests; had a friend try uploading a 400k⁉ message chat, and to no ones surprise it crashed the app.

So before adding in any more features, I plan on a full re-write to split it up into a React frontend (still leveraging plot.ly) and a FastAPI back-end, containerized and deployed on something like Container service or Kubernetes, allowing the compute intensive parts to scale with demand and save that money - I might not be a student anymore but frugality is a learned habit I plan to keep 🤑.

Additional Thoughts / Feelings / Stories

There's still a lot of improvements to be made before I feel it's ready for primetime, but I'm excited to see just how far this project will go. Supporting more messengers, real-time views of your chats, and much much more. This is just the beginning.

I plan on doing a more thorough deep-dive once I've ironed out some of the bigger outstanding issues, so stay tuned!

Here's hoping that someday it will empower anyone to learn from their data 🐱‍🏍

DEV Community: Amir