This is not your fault, this is a bad interview + bad company.lucky you didn't get it, this is not a company you want to work for anyway.
From what you've described, this sort of interview falls under the category of brain teasing. Teasing a candidate without giving any hints is pointless, what they should have done is to work through the solution with you together instead of pressing you alone to get the right answer. No one in a work place solves brain teasing/algo problem, what people do most of the time is team work and collaboration, this is actually a fundamental flaw in most of modern technical interview where a candidate is tested against something which is not even close to the company's real work environment.
Next time when you come across a interview like this, ask them why they wouldn't want to provide any hints, what is the objective of the interview and what are you expecting to get out of the candidate.
We're a place where coders share, stay up-to-date and grow their careers.
We strive for transparency and don't collect excess data.