There was a wonderful post just now that I saw on HN as I was procrastinating writing this post. It was by Tim Bray, and it covered my favorite topic, which is just how batshit cloud services have gotten, this time Traffic Director on GCP. I don't have anything more to say about this that you haven't heard before, blah blah blah, complexity bad, blah blah, simplicity good.
Which leads me quite nicely into some clarity I've gotten over the last few weeks. After reducing the team down to just me, I spent the last month doing a lot of thinking, really trying to get my head around where Dark is, what I should be doing, priorities, etc.
One idea that I raised in an earlier post is that perhaps Dark should try to do less. Rather than being the be-all-and-end-all of how to build apps in the cloud, I speculated that maybe aim a little shorter and build perhaps Zapier-but-with-a-programming-language, or perhaps a really nice forms-library-with-a-programming-language, or whatever.
I talked to users, contributors, investors, friends, etc, and spent a lot of time thinking about this, and basically concluded "fuck that". Almost everyone involved with Dark, me included, gets excited about revolutionizing backend programming, and I concluded that taking a detour from there is not interesting to anyone.
Positioning
That isn't to say that there isn't value in better-zapier or better-google-forms or whatever, but that these are nice use cases of Dark, rather than the actual goal of Dark.
The root issue here is that Dark is hard to position, in a marketing sense. Startups and projects tend to work best when they start targeting a small niche, like "retail", or "low-budget fast fashion retail", or even better "low-budget fast fashion retail targeting urban women aged 24-32". So everyone wants to know "what niche is Dark really good at?" And unfortunately the answer to that is far less specific than I would like, as a lot of tech is designed to let you do whatever you can come up with. "Programming language but for retail" makes about as much sense as "Kafka but for retail" or "computers but for retail".
We actually tried to do something like this with Slackbots, you may remember. We tried to get 300 Slackbots written in a month or so at one point, and I think we got around 15 written. The problem is that people just don't have that many Slackbots to write in a particular week or month. And when they do, they have the same problems that other people have, which is that the thing you want to connect your Slackbot to can be just about anything.
So we've struggled with positioning, and one of our big problems has been telling people what it is that we do.
Current State of Dark
Currently, I believe that Dark is a pretty effective MVP, but is actually further from product-market fit than I initially thought. The reason for this is that features that Dark needs to be successful are often hidden below either product or technical debt. We've gotten some very effective proof points that Dark is on the right track, in particular that the core things that I think make Dark great (Deployless, Trace-driven development, and the lack of infrastructure) are things that people love.
Roadmap
The connection between the MVP and the positioning might not be immediately obvious, so let me spell it out. Our positioning tells people that Dark is a particular thing, namely the easiest way to code backends. And then they try Dark and discover that that is not the case. And then they wander away.
I think both positioning and the product are missing the same thing, which is a sense of where Dark is going. To illustrate what I mean, here's an extremely rough draft of how I think I should position Dark:
Dark is trying to be the easiest way to build an API. It's experimental, shiny, with interesting features, and very very incomplete. Here's a list of what's interesting. Here's a list of what you can do with it. And here's a list of what sucks about it.
Then the website would, for each feature/use-case discussed, include the relevant parts of the roadmap right there. For example "Dark is currently good for teams of two or fewer. To handle teams of 10, we'll need Operational Transforms, global feature flags, a history of changes made, and code review".
Not coincidentally, this same problem (the lack of where we are going) has come up in the two contributor meetups we've had. People are eager to help, but unclear of how exactly to do so, beyond the fairly chore-like issues in the issue tracker. So a roadmap should help there too. Unsurprisingly, this was also a problem for the engineering team before.
Next steps
So I'm going to try to produce a roadmap over the next few months. I think a good way to approach this is to try to write some app in DarkV2, without actually implementing any of it, just essentially writing out the code in a doc. Then, as I "use" the DarkV2, write the spec explaining what exactly these features are, how they work, etc, focusing especially on getting enough detail to outline the difficult bits.
The aim is to say where Dark is going, so that we can say how we are going to get there, and present that in a useful form for users, contributors, etc.
The difficult bits here refers specifically to the things that we've learned in Dark so far. Like, how the HTTP "framework" is inflexible, or that it sucks that you can't write your own types, or that Dark doesn't have GraphQL built in, or that we're relying on traces in places where a type-system would be much more useful.
And in the meantime, I'm going to be doing a bunch of learning about various technologies (eg GraphQL, Rust) - as I'll be replacing large chunks of the product, now is as good a time as any to think about their implementation.
You can sign up for Dark here, and check out our progress in these features in our contributor Slack or by watching our GitHub repo. Comment here or on Twitter.
Top comments (3)
Glad to hear you say this as I was worried we'd lose such an interesting future in backend development.
Hi Paul,
I signed up for the beta and tried out the tutorial. I also had a look at the create a blog tutorial canvas. Here are a few thoughts:
Thanks for the thoughtful writeup! Planning on fixing 1, 3, 4 and 5. I'll take a look at how to simplify 6 also!
I really like the idea of a span of traces. Also, will be adding enums in the future, those are really important for exactly the reason you state!
There isn't really a comparison with low code, as it's not really intended for the same audience. Will think about it though.