This past week has been utter hell.
For whatever reason, last weekend, an insane amount of OpenAI API calls started cropping up. Like $200 a day. For the record, the usage is usually only about $5 a day. One day it got up to $1000.
Thus, pretty much this entire week was dedicated to this one issue.
We had to brainstorm solutions and do a ton of investigation, but my first hunch from the very start was having a malicious user just racking up usage because it was definitely unnatural. No way it was weekend traffic. That's millions upon millions of tokens.
Eventually after looking at the Kubernetes logs on GCP, I noticed that there was an error message regarding websocket connections closing, and there had been an issue with websockets disconnecting every 30 seconds, so I thought it might be related. However the other team added in logging via Mixpanel and started tying the issue to one account, which I then deleted from our database and it quickly went away (for the most part). This weekend, it was still being pretty crazy but not as bad as during the week itself.
I had tried myself to add Mixpanel tracking and token limits, but it was hard for me because I was sick from Tuesday to Thursday unfortunately.
I am hoping that we're able to launch as soon as possible because our project is hinging on investors for funding. I may have to take a temporary pay cut until we do get funding.
Well, anyway, until next Friday, cheers.
Top comments (0)