I have taken a break from working intensely on tgo for the last four days. Prior to this, I worked for probably three weeks straight, every single day, including Sundays, just getting this engine going. I pushed it to a point where things were stable, and then, as happens with these massive architectural shifts, things became really unstable again. So I pulled back, put in the work to make it much more stable, and took a breather.
But now I’m back, and I’m making a decision to try a new approach for Lodash and similar legacy libraries.
Why Don't I Just Hardcode This?
At the end of the day, we are going to have to inject custom typing for some of these foundational libraries. These are legacy JavaScript libraries. They are pretty stable, but because of their JS legacy, they aren't updated constantly, and I don't anticipate them expanding much beyond standard bug fixes. I think I'm going to try just adding typing.
Let's call them typing hints
The frustrating part is that I just want to actually use this damn tool I'm building, and I am being bottlenecked by getting stuck on compiling these legacy libraries. (Very important, yes, but trying to get this compiler to be "super smart and just figure it out", is probably the wrong approach. 3 weeks in on ONE..... ONE library, and I'm still not done. That 3 weeks could have been typing the entire codebase and submitting it.... a better idea, but one I'm saving to see if a few typing hints won't fix it.
Why I Refuse to Settle for any
No one is surprised that compiling dynamic JavaScript into strict, native Go is a pain in the ass. The big issue with my current approach is that it is an absolute whack-a-mole game trying to get the dynamic and specific typing working properly.
I could take the easy way out. If I just said, “Oh, make it all dynamic, just use any types,” it probably wouldn't be that bad to compile.
But I would lose all the goodness out of the system. In something as ubiquitous as Lodash, I love the idea of it being actually strictly typed on the inside so we get some serious, native speed out of it. If you just dump it into an any type, you are not getting the fullness of what tgo can do except in your own code. (So you better roll your own lodash/functional paradigm functions to get speed). I want native execution speed, and that requires types.
Injected Typings and Definitely Typed
What I am trying to do now is inject custom typings directly into the deep internals of these libraries.
I already have the Definitely Typed public interfaces working perfectly, so it can apply types to the high-level public API. Double thumbs up there. But getting there required me to actually put in a pull request to the Definitely Typed repository. Through this work, I discovered four actual functions, one of them being conforms, an assertion about the structure and shape of an object, that didn't actually have an export on their types.
Funny enough, I know I used conforms in a previous project, but we were writing in standard JavaScript at the time, so there was no compiler complaining about the missing type export. I submitted the PR, and that fix is on its way, so we'll see there.
But the more important focus is my decision to try injecting typings into some key internal, generic types to see if we can gain some ground. Lodash is heavily reliant on generic, low-level, reusable utility types like cache functions and hash-map structures that have zero typing information. Figuring out what those types are is a relentless game of whack-a-mole.
The AI CLI Illusion
This whack-a-mole game led to a harsh realization about relying on large codebases built by AI.
I didn't realize how much of the compilation errors were actually being hidden from me by the CLI user interface that AI built for the project. On the surface, it looked fine. The tests to see if the types and the code worked were passing. But when we hit the massive compilation process for an entire library, the CLI claimed that only 50 out of 300 entry points were failing.
In reality, all that meant was that the compiler was able to write out the Go file for those entry points. It did not mean those files would actually compile or run.
There was total confusion on what exactly the tool was reporting. It looked as though there were specific errors on one file or the other, but it was masking the truth. Eventually, I hit a breaking point: F that. You need to tell me exactly what endpoints are actually affected and print out the top 100 compilation errors so I can systematically look at it and build an approach.
When I finally exposed the raw data, I saw thousands of compilation errors spanning across the board.
Worst Case Scenario, Fix the Origin
Seeing those thousands of errors made me reconsider the approach entirely.
At this point, I could probably ask AI to write a script that calls an LLM with specific prompting to simply convert the entirety of Lodash into native TypeScript. Honestly, I could probably execute that faster than playing whack-a-mole with injected typings. Just like that, it would compile with strict types. I could even put in a pull request to open-source that TypeScript rewrite.
I am going to try the injected typings route one more time to see if I can get it to compile quickly. But if it stalls out again, I might pivot.
The Ultimate Goal: Zero-Friction Linking
All of this grueling work is serving one master objective: Absolute Zero Friction.
I have a really high standard for developer tools. The day that a toolchain becomes too much of a pain in the ass to maintain, I won't use it. I am trying to make tgo so seamless that a developer just types npm install lodash, and that is it. You don't have to do anything else. You don't have to worry about the underlying compilation.
And right now, the way our linking engine works is getting pretty amazing, I'm not gonna lie.
It automatically finds the source code and the exact version for your linked libraries. (If it can't find it natively, there's a clean method to inject that information). It goes out, pulls down the source code, converts it into Go, compiles it, and caches it. When you do follow-on compilations, it doesn't have to pull it down or convert it again. It just links it seamlessly, and it absolutely rocks.
I'm fighting through the legacy code right now so that when you run your builds, you get nothing but pure, optimized, native speed.
Top comments (0)