unsolvable problem alert
It's always news to me when I encounter a dev challenge I'm unable to surmount. It used to be unusual but the last two times have shown a pattern: it rears its head whenever I'm with undocumented tech
In this episode, I set out to build an auto documentation generator in php. Naturally, I leant towards psalm since suphle already uses it during server build to guarantee absence of type errors. Since this isn't documented anywhere, I combined several llms to tweak and fine tune the component until we successfully fabricated the tests, templates, classes and docs supposed to make this work. Took maybe a week
I didn't think more of it until time came to run the tests. After fixing the usual minute errors hindering system core from running, php memory allocation started exploding. It got so bad that it froze my system twice. Turns out that each time I hand a method over to psalm to analyse and give back its shape for documentation, it starts indexing the entire codebase anyway. No amount of config restricts it from crawling up my tremendous vendor folder. I opened an issue with them https://github.com/vimeo/psalm/issues/11871, that has remained unattended to till date
Decided to go hardcore and reverse engineer it. I managed to bypass the memory explosion but after some more days, I realised it wasn't sustainable. All their classes reside in an internal namespace. Each object expects some other mysterious dependency to already exist in some predefined state. Classes are marked final and methods span 100/120+ lines at times. I already updated my composer dependencies cuz of a tight version constraint of theirs. If I eventually figured out the very specific steps to scan just the one method return value, it'll be akin to walking on eggshells. One update and I'm right back to square one
So this is officially my unsolvable problem. I had to give up and pivot to a different strategy. I overhauled the psalm abstractions by introducing ast/phpParser. Even though I don't know what specific methods and types in their library to leverage, I now know enough about the flow to authoritatively guide llms on how it should work. I hate to say it but I think Claude was most impressive in its grasp of the situation. My problem with it is that I always max out the tokens after literally 3/4 exchanges max. Luckily, I had a summary from past llms already handy
Several hours later, it's as airtight as I want it. I'm yet to run the tests. But this isn't a victory lap. I've mused that it might very well be as much of a dead end as the psalm method was
Unrelated but you get the feeling that I'm a decent dev. Even though I didn't work on this daily for the weeks I've been stuck on it, it's humbling that folks like linus and brendan eich were able to churn out git and javascript in 10 days without stackoverflow or llms. Tbf I built suphle v1 before llms as well. Took 3 years but yea, me self no small
Top comments (0)