DEV Community: Darklang

Leaving OCaml

Paul Biggar — Mon, 02 Nov 2020 14:08:26 +0000

I built the first demo of Dark in Python, in about two weeks. A few months later when I started productizing it, I rebuilt it in OCaml. Back in 2017, when I was considering the language and platform to use for Dark, OCaml was extremely compelling:

it’s a high-level language with static types, so easy to make large scale changes as we figure out what the language/product was
you mostly model data with sum types, which in my mind are the best way to model data
it’s very similar to the language I wanted to build (in particular, we could reuse built-in immutable data structures for Dark’s values)
it had a reputation for being high-performance, which meant that we could write an interpreter for Dark and not have it be terribly slow (vs writing an interpreter in python, which might be too slow)

Unfortunately, as we’ve built Dark we’ve run into significant problems that have made it challenging to build in OCaml.

Lack of libraries

When you bet on an off-mainstream language, one of the things you accept is that many libraries are not going to be available. When there is a small community, often there aren’t enough people working in the language to make important libraries. This is especially true if few people are building business applications.

In OCaml there are many high quality libraries, especially for data structures and data manipulation. The annual Jane Street code dump has been quite useful and very high quality. However, we really felt the lack of several libraries. The most obvious of these is that we had to build a Unicode string library ourselves (built on top of the very impressive OCaml Unicode libraries built by Daniel Bünzli), but we needed many more libraries than that.

The lack of an SDK for Google Cloud has affected us greatly. When you’re searching for product-market fit, you do the simplest, easiest thing. If you lack a good SDK for your cloud provider, the simplest, easiest thing is often a terrible architectural choice. We’ve built our own queue on top of our database rather than using the production-quality cloud queues available on GCP. Similarly, we barely use the Cloud Storage (GCP’s version of S3), because we initially put things in the database because it was easier. We’ve built 3 services, 2 in Rust, and 1 in Go, to workaround the challenges we’ve faced.

The biggest challenge here is our use of Postgres. Postgres is a great database and we’re big fans, but Cloud SQL is not a great hosted database. GCP’s position is that Cloud SQL is there to tick a box and we should be using Cloud Spanner. I would love to switch to Cloud Spanner, but we have no driver for it in OCaml. Given the Postgres driver in OCaml is not particularly mature, it’s hard to expect that a Cloud Spanner driver would exist, and indeed it doesn’t. We’ve had to contribute to the OCaml Postgres driver, and some parts of our codebase have been well and truly mangled when working around features not supported in that driver.

We’ve also suffered from a lack of a high-level, production web stack (there are low-level stacks with good reputations that I’ve struggled to use, and a few new ones out there that look good), in particular lacking a user authentication module. We’ve been using Auth0 to work around this for now, which has more moving pieces than I’d like, and a shockingly high cost (our 7000 users, most of whom never log in, costs us over $500/mo).

We’ve worked around other missing vendor SDKs by calling their HTTP endpoints directly and that’s been mostly fine. However, for libraries like encryption we don’t have that option — we hacked around a missing encryption library, but decided not to ship it to production until we audited it for security (which was never actually worth the cost).

At CircleCI, we bet on Clojure. That was also a non-mainstream language, but its ability to call Java SDKs meant we had a mature cloud library, which was essential for building CircleCI. Of course, in OCaml we could call C libraries (and even Rust libraries, perhaps), but it doesn’t match having native libraries we can call directly.

Learnability

I’m mostly in the camp that anyone can learn any language, but I saw a team struggle with OCaml, and for good reason. Language tutorials are extremely poor in OCaml compared to other languages; they’re mostly lecture notes from academic courses.

The compiler isn’t particularly helpful, certainly compared to Rust or Elm (both of which have been in our stack at one point). Often it gives no information about an error. Syntax errors typically say “Syntax error”; though it will try to give a good error for a mismatched brace, often incorrectly. Type errors can be a real burden to read, even after 3 years of experience with it.

The docs in OCaml are often challenging to find. The Jane Street docs have improved significantly in the last few years, but it can be a challenge to even figure out what functions are available in a particular module for most libraries. Compare to the excellent docs.rs in Rust, which has comprehensive API docs for every package in Rust.

One of the ways I personally struggled in OCaml is around Lwt. Lwt is (one of!) OCaml's async implementations. I couldn't figure it out several years ago and so just built a single-threaded server. The amount of workarounds and downtime we've suffered from that single decision is immense. A tutorial around building high-performance (or even medium performance!) web servers would be very valuable.

Tooling

Tooling is something I read would be good in OCaml. I remember reading there was a debugger that could go back in time! I don’t know where that’s gone but I’ve never heard of anyone using it.

We have struggled to make editor tooling work for us. This is partially because we also use ReasonML and this seems to break things. Unfortunately, this is common in programming, but even more so in small communities: you might be the first person to ever try to use a particular configuration.

Finally, the disconnect between the various tools is immense. You need to understand Opam, Dune, and Esy, to be able to get something working (you could also do it without Esy and just rely on Opam, but that’s much worse). I talked about a bunch of these challenges here.

Language problems

Multicore is coming Any Day Now™️, and while this wasn’t a huge deal for us, it was annoying.

Minor annoyances

One of my biggest annoyances was how often OCaml folks talk about Fancy Type System problems, instead of how to actually build products and applications. In other communities for similar languages (ReasonML, Elm, F#), people talk about building apps and solving their problems. In OCaml, it feels like people spend an awful lot of time discussing Functors. It’s not quite at the level that I perceived in the Haskell world, but it pointed out that the people building the core of the ecosystem do not have the same problems that I do (which is building web-y stuff).

Was OCaml the wrong choice?

I honestly think OCaml was a great choice at the start. Being able to quickly and safely make large-scale changes to your app is something that statically-typed functional languages excel at. I’m happy that we made the choice, and in retrospect, it still seems like the best choice of those we had at the time.

What’s next?

I’m working on building the next version of the backend. We have about 20k lines to be replaced, and they’ll be rewritten in a new language while keeping the semantics the same. I plan to leave keep the frontend in ReasonML: it doesn’t suffer from the same library problems as it can interface nicely to JS, and it’s nearly 50k lines of code so it would be a much bigger undertaking.

I’ll write in a few days about what we chose instead.

You can sign up for Dark here. For more info on Dark, follow our RSS, follow us (or me) on Twitter, join our Slack Community, or watch our GitHub repo.

Dark v2 Roadmap

Paul Biggar — Mon, 12 Oct 2020 13:21:25 +0000

Dark v2 Roadmap

tl;dr there’s a Dark v2 roadmap now.

There are a number of things missing for Dark to be able to have product market fit:

Dark lacks SDKs for 3rd party APIs
Dark lacks a good user account/authentication story

While these are the major issues, solving these requires some more fundamental changes. As well as simply making the package manager available, we also need to improve on the tools that developers have for building packages, everything from how packages coexist with other functions, the types available to packages, and how to support collaboration on those packages.

Similarly, there is a bunch of tooling needed before we can build a good user authentication story. Dark needs to ability to define HTTP middleware (we need to decompose the existing HTTP framework into middleware), as well as better types. Each of these changes then depends on further changes — you need a good language to have good APIs.

We have a lot of information on how exactly these features need to work — Dark users have been giving us feedback for quite some time. In order to turn this feedback and these needs into actionable changes, I’ve started putting together the Dark v2 roadmap. Note that it’s early and there’s still a lot to be written.

There’s a lot to build, so specing it out in the roadmap will save a lot of time. I hope it will also allow Dark developers and potential developers to give feedback on the product direction, as well as contribute features if they like.

The roadmap is structured around identifying the problems in Dark v1 and their fixes in Dark v2. Note that I call them Dark v1 and Dark v2, but I don’t expect a hard break, it will sort of migrate it’s way there over time.

The problems and fixes are across the product. One problem is that undo is slow, another is that you can’t put a minus sign in front of an integer, another is that we need to define how namespaces will work in the package manager. The fixes range from adding tooltips in the UI, to adding a type-checker, to making the package manager public.

Please check it out! Feedback, thoughts and words of encouragement are very much appreciated.

You can sign up for Dark here, and check out our progress in our contributor Slack or by watching our GitHub repo. Follow Dark (or me) on Twitter, or follow the blog using RSS.

FizzBoom benchmark (call for participation)

Paul Biggar — Wed, 23 Sep 2020 15:44:12 +0000

I’ve been working recently on a benchmark, to try and see how to get the most performance from the Dark backend. I’ve reimplemented the core of Dark in various languages and web frameworks, notably in OCaml, F# and Rust.

As a reminder, Dark is a language and platform for building backend web services and APIs. Implementation-wise, it’s basically an interpreter hooked to a webserver and a DB. The language is a statically-typed functional, immutable language which is garbage collected.

Dark users can write arbitrary code that runs on our server, including making HTTP calls to slow and badly-behaving 3rd-party web servers. This means we need to efficiently support both computation and IO on the server. This benchmark is meant to measure that.

FizzBoom

FuzzBuzz is a well-known interview question, that may or may not have been appropriate in a bygone era, but that remains in existence today due to interviews that have not moved on. It asks you to list 100 numbers, and if they are divisible by 3 or 5 you print “fizz” or “buzz” respectively.

FizzBoom is the same benchmark, except instead of printing “fizzbuzz” when a number is divisible by both 3 and 5, you instead make a HTTP call to a local server which takes 1 second to respond. Here’s what this looks like in Dark:

let range = List::range 1 100
List.map range \i ->
                  if i % 15 == 0
                  then
                    HttpClient::get "http://localhost:8000/delay/1"
                  else if i % 3 == 0
                       then "Fizz"
                       else if i % 5 == 0
                            then "Buzz"
                            else toString i

Call for participation

Typically, someone builds benchmarks and then releases them to the wild. Invariably, they’ve overlooked something, and hordes of language advocates decry how it’s unfair. To avoid this unfairness, I’d like to create an opportunity for language advocates to improve the benchmarks before I release them. I’ll discuss the goals of this below, but feel free to jump right to the issues if you prefer.

The benchmark measures two numbers, calculated using wrk:

requests per second for HTTP calls to /fizzbuzz, which returns FuzzBuzz as JSON
requests per second for HTTP calls to /fuzzboom, which return FizzBoom as JSON.

Preliminary numbers (thanks Chris for reporting these) indicate F# is great (25k req/s) and Rust and OCaml are doing ok (16k req/s each), for FizzBuzz.

    Benchmark name                Req/s
    ---------------------------------------------------------
    fsharp-giraffe:               24962.95
    fsharp-giraffe-async:         19476.78
    fsharp-suave-async:            1147.71
    fsharp-suave-partial-async:   Skipping broken benchmark
    ocaml-httpaf:                 14034.62
    ocaml-httpaf-lwt:             14158.74
    rust-hyper:                   15985.69
    rust-hyper-async:             Skipping broken benchmark

However, it also shows that we’re not doing async right on any platform — FizzBoom languishes at 1 req/s on all platforms. Obviously, this is because the code I wrote doesn’t work, and is not actually a reflection on the languages and frameworks.

    Benchmark name                Req/s
    --------------------------------------------------------
    fsharp-giraffe:               1.00
    fsharp-giraffe-async:         8.99
    fsharp-suave-async:           1.00
    fsharp-suave-partial-async:   Skipping broken benchmark
    ocaml-httpaf:                 0.10
    ocaml-httpaf-lwt:             0.10
    rust-hyper:                   Invalid fizzboom output
    rust-hyper-async:             Skipping broken benchmark

Improving your favorite language’s performance

If you’re worried about your language doing well in the benchmark, or are simply looking to help, there are a number of things you can do:

optimize your language’s web server: I may have used a poorly performing web server, have it in a poor configuration, or have hooked things up poorly
fix your language’s async benchmark: when making a HTTP call to a 3rd-party webserver, the server should free the CPU to handle other requests while the IO is running. I don’t have that working correctly in any language yet (unsure why this isn’t trivial in all platforms, but there we are)
optimize your language’s build performance: fix the build settings so that it’s being optimized to the best of its ability.

I’m also interested, but very cautiously, in improving the implementation of the interpreters. The prime directive for the interpreters is that they’re easy to modify and extend: Dark is a language going under much change, so rewriting the interpreters with a JIT, or assembly, or peephole optimizations, is not something I’m interested in. But small implementation changes that have big wins are valuable, whether they’re applicable to all languages or show off specific features of your favorite language. All the same, I’m going to be quite conservative on this, I don’t want to turn this into a game where we write non-idiomatic code to squeeze out a win that no-one would want to maintain.

If you’d like to help, I’ve filed the issues above, and written down some rules to keep the benchmark fair. I hope to release some benchmark results soon, once I’ve got the majority of them working.

You can sign up for Dark here, and check out our progress in our contributor Slack or by watching our GitHub repo. Follow Dark (or me) on Twitter, or follow the blog using RSS.

Adventures in Async

Paul Biggar — Thu, 17 Sep 2020 20:40:08 +0000

Adventures in Async

I was very recently the holder of three opinions:

Rust is magic fairy dust that can fix all my problems
Async is magic fairy dust that can fix all my problems
GraphQL is magic fairy dust that can fix all my problems

Unsurprisingly, all of these appear to be wrong. This post is about the first two.

In the journey to get to product/market fit with Dark, I’m taking a slightly different strategy than we took before, which I’m calling “Hard Things First”. Repeatedly with Dark we spent significant time coming up with hacks to work around previous hacks which themselves were working around previous hacks. That made for a codebase where people can only contribute is well-defined ways, as going outside the box was to invite madness. Or more specifically, to need to have the entire history of the codebase and company and roadmap in your head to understand why the code is like that.

So instead, I’m looking at doing the Hard Things First. Dark has a few problems in its server-side implementation, and those need to be fixed. We long speculated that we’d be able to hack them for now and do a Big Rewrite Later, with the copious resources that we’d have after the Series A. But alas, that is not how things are going, so I now need to figure out some of these Hard Things, and the first question is whether to stay in OCaml or switch to something else.

Staying in OCaml

If we stay in OCaml, I need to figure out how to solve a number of key problems, the biggest being how to not hang the server when making a slow HTTPClient::get request. The Dark web server is currently synchronous, and so long or slow requests--at sufficient volume--can cause operational issues for us. Solving that is something we aggressively put off; but the first of the Hard Things First to address. And that's where async comes in.

Quick async recap

As a quick introduction, the word async describes a way of allowing servers respond to many more requests by taking advantage of non-blocking IO. In the old days, servers used one thread to respond to each request. Now, servers commonly use an async implementation where a single thread can handle many many requests.

They do so by handling the requests in a simple loop. Each request is added to a queue when it comes in, and the request at the front of the queue is run by the thread. If the handler does some IO (perhaps talking to the DB or to a HTTP API), that thread will stop working on that request, and move onto the next request in the queue. When the IO finishes, it is re-added to the queue, where the thread will come for it shortly.

This is sort of like context switching between threads, but much lower overhead as you can switch between items in the queue much more cheaply than switching between threads. A single thread can handle dozens or hundreds of requests at once (so long as they are mostly IO-bound), so you can process many more requests than using simple multi-threading (and of course, you can still be multi-threaded in a multi-core situation, running an async server on each thread).

If you enjoyed that, you’ll probably enjoy and learn a lot from this post.

Back to OCaml

In OCaml, there’s two competing async implementations, Lwt and (the aptly named) Async. The programming model isn’t terrible since OCaml 4.08, when they added a let* keyword. This is roughly akin to the await keyword in JS, Python and Rust. So you take your synchronous code:

    let sync_function () : int =
      6

    let x = sync_function () in
    x + 2

and rewrite it asynchronously

    let async_function () : int Lwt.t =
      Lwt.return 6

    let* x = async_function () in
    Lwt.return (x + 2)

Lwt.t here is a promise, and it's the same concept as a Promise in JS, and Async<...> in Rust and F#.

The difference between the Lwt and Async libraries seems to be that Lwt is greedy (it will keep going on the same request if it can, that is, if the promise resolves fast enough) and Async is not (it will spread the work around if it can). So, roughly speaking, Lwt prioritizes latency and Async prioritizes fairness and preventing requests from being starved.

Other OCaml improvements

Of course, the lack of async isn’t the only problem in the Dark codebase. There are many things in our codebase that make it attractive to look at redoing the server-side implementation in another language.

However, having looked through the backend codebase recently, I think most of it can be solved by refactoring and using some of the tools that have come along in the last few years. In particular, Caqti solves some of the needs that are not solved by using Postgresql-ocaml directly (such as connection pooling). Slih has come along to make it much nicer to write web services in OCaml. httpaf and H2 look to solve some of the core performance problems in CoHttp. And OCaml-graphql-server provides a nice graphql server in OCaml.

The other stuff that needs to be worked on to keep in OCaml is the tooling, but I think a lot of the problem comes from how we’ve mashed everything together, and a little bit of performance optimization in our build scripts might solve a lot of problems.

Rust and F#

I spent some time rewriting the core of Dark in Rust and F#. Not the full thing, but just enough to connect a server to a Dark interpreter and calculate Fizzbuzz. The goal is to get a feel for the ecosystem, programming model, and of course the performance for asynchronous workloads like people build on Dark. Early learnings indicate that magical fairy dust does not exist.

The first learning, and I’ll probably write some more about this later, is that coding in Rust is hard. Like, much harder than I expected. And using async in Rust is exponentially harder than writing synchronous Rust, to the point that I’m not even sure that it’s a good idea for anyone to do it (again, more on this in a future post). I had not realized how much the garbage collector does for you in managed languages, and how much complexity managing your own memory adds in an asynchronous server.

By contrast, rewriting the core of Dark in F# was very simple, and adapting it to use async was also extremely straightforward. This comes from someone who doesn’t know anything about the dotnet ecosystem, but who does of course know OCaml, which is about 95% the same at F#.

The code is at https://github.com/darklang/sync-async-benchmarks, which I hope to turn into benchmarks. I hope they’ll help inform my decision about which way to move forward. I’ve been learning a lot from the experiments with Rust and F#, and I’ve been using the experience to start working on a roadmap/spec for the next version of Dark, which I’ll talk about soon.

As you can imagine, I’m eager to stop experimenting and make some decisions, and I’m looking forward to getting back to moving the Dark language and implementation forward.

You can sign up for Dark here, and check out our progress in our contributor Slack or by watching our GitHub repo. Follow Dark (or me) on Twitter, or follow the blog using RSS.

How to add a new feature to Dark

Paul Biggar — Tue, 01 Sep 2020 19:53:28 +0000

On Saturday, we had our third contributor meetup, this one was organized by Shahriyar Nasir (thanks Shah!). This time, we did mob coding to add a Tuple type to Dark, and we recorded the whole thing.

A tuple looks like this:

let x = (somevar, "a string", 567)

A tuple allows you to combine multiple values of different types. In the video, we added Tuples to the editor and frontend, including in particular the keystrokes to add, update, and delete tuples. Shah followed up later to add backend support. We hope to get this merged soon.

The video is about two hours, which is about how long it takes to add a language feature to the frontend (it probably takes a similar amount of time to add a feature to the backend). This was straightforward as Tuples have a very similar interaction model to Lists - hopefully other language improvements won't take too much longer.

Note that if you're interested in contributing other language features, the issues have a good list of what we need.

Now that tuples exist in the language, we still have make them more usable. We need functions to get the contents, such as Tuple2::getFirst, Tuple2::getSecond, etc. This also highlights that we need to support destructuring: while we currently support destructuring some expression types in pattern matches, we don't support that in let statements. We support:

match someNumber
  Ok 5 -> 6
  Ok 6 -> 7
  Ok num -> num + 1
  Err msg -> 0

But we don't support:

let Ok num = someNumber

There are quite a lot of functions in Dark that could return Tuples, where we've hacked support using other methods. For example, request.headers uses a dictionary (as does the header parameter in HttpClient::get), DB::getAllWithKeys_v2 returns a dictionary, and List::zip returns a list of two-element arrays. These would much more naturally use tuples.

Overall, this is part of fleshing out the language, in order to allow us to build new features without having to hack around an incomplete language. We'll be doing more of this in the next few months.

You can sign up for Dark here, and check out our progress in these features in our contributor Slack or by watching our GitHub repo. Follow Dark on Twitter, or follow the blog using RSS.

First thoughts on Rust vs OCaml

Paul Biggar — Thu, 20 Aug 2020 13:18:15 +0000

I'm about two weeks into Rust now, so this feels like a good time to write a critique, before I get Stockholm Syndrome'd.

My main motivation in learning Rust is that I have to maintain some of Dark's Rust code. There was a recent outage related to that code, and I had to learn on the fly, so better to actually know what I'm looking at.

I've also been dreaming of rewriting Dark in Rust for quite some time, largely due to frustrations with OCaml as well as some excellent marketing by the Rust community. I'm trying to evaluate whether this is a good idea, and if so, trying to figure out how to do it in a way that makes sense (which is to say, gradually).

By the way, Dark is platform for building backend apps; in practice it's a HTTP server and database (which comprise the "Dark platform") which connect to an interpreter for the Dark programming language, which you edit using the Dark editor. The interpreter is the core of the language, it runs in Kubernetes in Google Cloud, as well as compiled to JS in the browser. Today, almost everything is implemented in OCaml.

Initial impressions

So far I've spent a few weeks learning Rust, and I have to say, I like lots of things about Rust but I definitely don't love it yet.

The lens I use when talking about programming languages is Accidental Complexity, which is basically what Dark is all about (removing it, that is). So when I critique something, it's largely based on whatever bullshit hoops I feel it makes me jump through.

I'll be complaining about both OCaml and Rust below, so if you're a fan of either, prepare yourself. Remember, all programming languages suck, especially whichever one is your favorite.

What's to like about Rust (and dislike about OCaml)

There's a few things that Rust does very well, that OCaml really does not. I had hoped that OCaml would improve over the years, and for sure there's progress being made, if slow. But once you come face to face with a language that does it well, the difference is really stark!

Community

People say Rust is a fringe language, but as someone who's been coding in a truly fringe language (OCaml, and Clojure before that), Rust feels huge. There's tons and tons of ways to learn: an impressive tutorial (and lots of community tutorials as well), multiple books, StackOverflow questions, blog posts, videos. Not only that, but there's lots aimed at beginners; often it feels like OCaml blog posts are aimed at hardened professionals or academics. I recall finding it very very hard to get into, and especially hard to figure out how to write idiomatic OCaml, which I have not found with Rust at all.

Libraries

It's amazing - Rust has libraries for everything!! I've really been struggling with the dearth of libraries in OCaml. A year or two ago we were chatting to Google Cloud about deficiencies in Cloud SQL Postgres, and got told that Cloud SQL is only really there to tick boxes, we should move to Spanner instead. Except OCaml has no Spanner library, so we're stuck on Cloud SQL. Repeat this 50 times for 50 different vendors and services, and you start to get frustrated.

Tooling - building

The tooling in Rust is also fabulous. In particular, Cargo is probably the best build system I've ever used. The fact that they have integrated the package manager, the build system, and the compiler, and then it all just works? Magic!

By contrast, in OCaml you have opam, esy, dune, and OCaml itself all doing part of the job. They sorta kinda mostly work well together, until they don't. And as you wire them together, you really need to understand what each tool does, where it starts and stops, and how they integrate. Honestly, not fun at all.

The frontend is also confusing: there's the Bucklescript stuff, the ReasonML stuff, the OCaml compiler stuff, the Javascript stuff. But recently a nice improvement - they made it just one thing! So, some improvements there at least!

Tooling - editor

The editor tooling is also great in Rust. The Rust Language Server "just works" (at least in VS Code), whereas I've had a lot of trouble with setting up OCaml editor integration. I once had it finally working really well, and then I changed something in my OCaml config and have been unable to bring it back.

Tooling musing

I'm going to put my Dark biases to work here and speculate that all this amazingness is because Rust is mostly an integrated system: the same people are building all of these tools (if not the same exact humans, at least a group of people working towards the same goals). Whereas it doesn't feel that way in OCaml at all.

OCaml feels like a completely separate camps, where Facebook, Inria, Jane Street, and OCaml Labs, all sorta work on their own stuff and that few if any of them talk to each other. That's an outsider view (one I've voiced before) and possibly wrong, but if you folks do speak, maybe integrate a few of these projects and lower the cognitive overhead of learning all this stuff? Wouldn't it be great if there was a single tool as simple and powerful as Cargo for the entire OCaml/Reason ecosystem?

Macros

Macros in Rust are great! OCaml has PPXes, which are separate binaries that you build using the OCaml compiler toolkit. They have a very high barrier to entry, and I've never built one, and really struggled to even understand the ones I use. They also seem to be having a lot of trouble over the last few years.

Macros in Rust are built into the language. It feels a little like doing a regex over part of the syntax tree (with the same downsides, honestly), which is to say that they're pretty easy to get started and do quite powerful stuff with, even if they're not that powerful.

They do actually seem to lack some of the power of OCaml's PPXes. With OCaml's PPXes, I believe you can access the entirety of the compiler, with all of the upsides (do anything!) and downsides (compatibility between versions). However, with Rust I seem to have much less access - unless I'm mistaken I can't actually get it to parse type information for generics that I could then repurpose, which is frustrating.

They seem to be easier to write than Clojure's macros, though also less powerful. Given that Lisp/Clojure is entirely designed around treating code as data, and there's so little syntax, this makes sense. But still, pretty nice!

Aesthetics

Rust feels really well designed, which I expect is because they've aggressively pruned the parts of the language that don't make sense. So the Rust that I see today is supposably quite different from the Rust of 2010. Meanwhile, OCaml has a ton of stuff that they really should prune and have not.

Similarly, Rust the language is quite pretty, syntax-wise. Meanwhile, OCaml is so ugly that the community came up with a whole other syntax for it. This shouldn't be a big deal, but as someone responsible for getting folks up to speed on OCaml for a few years, this was not a fun experience.

OK, what sucks about Rust?

Apologies to the OCaml folks for being the victims of my frustrations so far; time to talk about Rust. First, some expectations: I've been working on Rust for two weeks, and a lot of that was doing the tutorial, worked examples, etc. But I do have a lot of experience with OCaml and Clojure, and also significant experience in C and C++. I've also been programming professionally for about 20 years, mostly in compilers and programming language implementations. So I feel Rust should be straightforward.

While we're talking about me, here's a fun anecdote about my first experience with Rust: In 2010, I started work at Mozilla. For my first day, they flew me to Whistler for their annual retreat, and I got to see Graydon Hoare present the thing he'd been working on for 4 years in secret: a new programming language called Rust! I excitedly bounded up to him to offer help: I know compilers, I love languages, I can help!! Unfortunately, though I tried I didn't get very far in helping, as I really couldn't get my head around the language that Rust was implemented in at the time, which was an esoteric academic language called "OCaml"!

Memory management

I was actually surprised at how little the actual memory management bothered me. I'm a big believer in garbage collection, and not having to think about memory, so I expected to hate this part of Rust, but it turns out it's kinda OK. You put everything in a Box::new (regular heap memory) or RC::new (reference counted memory) or Arc::new (reference counted memory suitable to be used concurrently in different threads), and then when they go out of scope they'll be cleaned up.

This actually seems kinda OK, especially when I compare it to writing C++ back in the day. I've only built the very small part of an interpreter so far, but I'm expecting that if I take care of scopes properly, the memory will just sort of manage itself? Not as good as garbage collection IMO, but not as bad as I'd heard.

Pattern matching

I was initially excited that Rust has pattern matching, since that's something every language should have (and maybe will soon). However, I pretty quickly learned to be less excited about this.

Let's quickly look at some interpreter code to see why.

type t =
  | EInteger of id * string
  | ELet of id * string * t * t
  | ELambda of id * (analysisID * string) list * t
  | EVariable of id * string
  | EFnCall of id * string * t list * sendToRail
[@@deriving show {with_path = false}, eq, ord, yojson {optional = true}

This is a cut down version of the original in OCaml.

#[derive(Debug)]
pub enum Expr_ {
  Let {
    var: String,
    rhs: Expr,
    body: Expr,
  },
  FnCall {
    name: FunctionDesc_,
    args: im::Vector<Expr>,
  },
  Lambda {
    params: im::Vector<String>,
    body: Expr,
  },
  Variable {
    name: String,
  },
  IntLiteral {
    val: i32,
  },
}

pub type Expr = Arc<Expr_>;
unsafe impl Send for Expr_ {}
unsafe impl Sync for Expr_ {}

So we can see the same type here in two different languages. In OCaml, you just specify the type. In Rust, you specify the type and also the garbage collection strategy - you can't just put an Expr in another Expr as you don't know the size of the Expr, and so need to wrap it in a Box, Rc or Arc. In C, you'd use a pointer, and really that's what we have here: both OCaml and Rust are effectively using a pointer to another Expr.

But what happens when we get to pattern matching? Well, here's how we do it in OCaml (full version here):

(function
 | state, [DList l; DBlock b] ->
     let f (dv : dval) : dval = Ast.execute_dblock ~state b [dv] in
        Dval.to_list (List.map ~f l)
 | args ->
     fail args

And in Rust? Here's how I'd like to do it:

    ("List", "map", 0), |args| match args {
      [ DList(list), DLambda(vars, body) ] =>
          DList(list.map(|val| execute_block(vars, body, [val])))
      args -> fail(args)
    })

However, I can't actually pattern match like this in Rust because I don't have DLists, I have Arc<Dlist>, and you can't pattern match through Arcs and Rcs and Boxes. (And also, I need to deal with ownership of basically everything here).

I eventually got this to work by doing:

match args.iter().map(|v| &(**v)).collect::<Vec<_>>().as_slice()

and then wrapped it in a macro to make it tolerable. Needless to say, this is quite frustrating.

Too many ways to do things

I remember when I first started Ruby; we were using JRuby in the early days of CircleCI, and also Mongo. Both of those were well supported, with many people using them. However, we seemed to be the first to ever use them together and I quickly put together this rule of thumb:

you can go off the beaten path once and be OK, but not twice.

In Rust, I'm seeing that there are many alternate paths, that aren't exactly off the beaten track, but that open multiple options and I'm starting to get worried. The most obvious ones being async/sync and Rc vs Arc.

I've already seen the seeds of Rc vs Arc. I'm using im, a library of immutable data structures (more on that later). I read that the idiomatic way to do composable errors in Rust is to use error-chain, which uses Arc. I was using im with Rc and that wasn't compatible with the Arcs in error-chain.

This seems to have come up before, as im actually has two libraries: im (which uses Arc) and im-rc (which uses Rc). The fact that there are two identical libraries with identical functionality, except that one uses Rc and the other uses Arc, seems to be a giant red flag that I feel is going to cause me a lot of trouble some day.

Similarly, it seems that sync and async Rust use entirely different standard libraries, which is also a big red flag. Do all flavors of im and Rc/Arc and sync/async play well together? No idea, but finding out is not an exciting prospect.

Immutable

Using im for data-structures is my 3rd "off the beaten track" thing in Rust. im is a library of immutable data structures, which is useful for - amongst other things - creating languages which are default immutable.

I initially thought Rust was immutable, but it is definitely not. While it has some amount of immutability built in, fundamentally you're acting on mutable values. It is super useful to have to ability to know that you can pass some to a function and it won't change, this is extremely wonderful.

However, when I compare this to the standard way to do things in immutable languages such as Clojure and OCaml, it has a big downside. Clojure and OCaml both use "purely functional data structures" which allow you to get new versions of data structures without destroying the old one.

For example, if I have a list and I want to do some operation on a slightly longer list, I can have a value that keeps the old list intact but also allows me to interact with the new slightly different list without a large performance cost. In Rust's default Vec, I'd have to do a clone of the old list for this sort of operation.

I find this super frustrating, and it feels like managing this is the cause of a lot of the boilerplate/accidental complexity in Rust.

A very common thing I'm doing is to create a new "scope" (or symbol table) in Dark, which is the old scope plus a few variables. In mutable languages, this is expensive in terms of performance, so language implementations often create a sort of stack of scopes to manage the performance implications. With immutability, you create a "copy" of the old scope for free, and then add your new values to it. When you're done, you just keep using the old scope, which hasn't actually changed.

So it's really quite frustrating to go from a language with immutability at its core, to one which has only some immutability. im handles some of the complexity, but I find myself constantly converting things to Vecs (as we saw above), which sorta defeats the purpose.

My point here is that I wish Rust was properly immutable, I guess. Hard to do in a systems language, but kinda frustrating.

Satisfying the compiler

I said I'd complain just a little about the type system (or "borrow checker" or whatever--are these the same thing or different? I'm not sure yet). Programming in Rust reminds me a lot of programming in C++: you add a const to one function, and then you have to follow that const around the entire codebase until you finally get to the place where you learn it actually can't be const, and so fuck you. Except in Rust this happens constantly. I'm hoping this will go away after I get used to it, but it's quite frustrating.

In OCaml, this experience is different: since basically everything is immutable you never get this, and you don't track your own pointers so everything is magic. Although I will say that very few times where something is actually wrong (usually the type inference does not infer using the same algorithm that a reasonable person would use), and the error messages in OCaml are very nearly useless.

Here's a nice metaphor: suppose you go to Chipotle every day. At Rust Chipotle, they have strict rules about the ingredients for your burrito. "White rice with medium salsa, sir? Absolutely not!". You see, medium salsa only goes with brown rice, and you also need to have beans or nothing works. Under no circumstances will they allow you to construct the burrito you think you want, no matter how much you think you want it.

Meanwhile, at OCaml Chipotle you can have whatever you like, and it always turns out awesome. But once a month you go for lunch and they'll refuse to make you a bowl, refuse to tell you why, and refuse to let you leave. And when you try to get help from a passerby after being trapped in the store, you realize there's nobody nearby who you can ask.

Macros

Macros were also in the "what's great about Rust" section. However, it feels like a code smell that I'm turning to macros so often. There's a ton of boiler plate, and I'm finding it hard to write reusable code using functions, what with all this ownership stuff. I expect I'll figure this out and won't need to lean on macros so much.

Interpreters & Compilers

Now to be fair, this might be a little bit of an unfair comparison. You see, OCaml is a language whose Raison D'etre is to build interpreters and compilers. I had heard that Rust is excellent at compilers too, so I was expecting it to be just as good. Maybe my brain is too warped by immutability - between Clojure and OCaml and Dark, I've barely actually coded in a mutable language in a decade.

Conclusion

Overall, Rust the language is frustrating but not terrible, while the tooling, community and ecosystem are truly excellent. Meanwhile, OCaml has poor tooling, poor ecosystem, and no community, while the language itself is truly excellent (except for the horrendous syntax, of course).

Since I've really hit a dead end with OCaml (multiple times even), I'm still hoping I'll get Rust Stockholm Syndrome. There doesn't really seem to be anywhere else to go.

Discuss on Twitter, Hacker News, or Medium.

You can sign up for Dark here, and check out our progress in these features in our contributor Slack or by watching our GitHub repo.

Directions and roadmaps

Paul Biggar — Tue, 11 Aug 2020 17:28:40 +0000

There was a wonderful post just now that I saw on HN as I was procrastinating writing this post. It was by Tim Bray, and it covered my favorite topic, which is just how batshit cloud services have gotten, this time Traffic Director on GCP. I don't have anything more to say about this that you haven't heard before, blah blah blah, complexity bad, blah blah, simplicity good.

Which leads me quite nicely into some clarity I've gotten over the last few weeks. After reducing the team down to just me, I spent the last month doing a lot of thinking, really trying to get my head around where Dark is, what I should be doing, priorities, etc.

One idea that I raised in an earlier post is that perhaps Dark should try to do less. Rather than being the be-all-and-end-all of how to build apps in the cloud, I speculated that maybe aim a little shorter and build perhaps Zapier-but-with-a-programming-language, or perhaps a really nice forms-library-with-a-programming-language, or whatever.

I talked to users, contributors, investors, friends, etc, and spent a lot of time thinking about this, and basically concluded "fuck that". Almost everyone involved with Dark, me included, gets excited about revolutionizing backend programming, and I concluded that taking a detour from there is not interesting to anyone.

Positioning

That isn't to say that there isn't value in better-zapier or better-google-forms or whatever, but that these are nice use cases of Dark, rather than the actual goal of Dark.

The root issue here is that Dark is hard to position, in a marketing sense. Startups and projects tend to work best when they start targeting a small niche, like "retail", or "low-budget fast fashion retail", or even better "low-budget fast fashion retail targeting urban women aged 24-32". So everyone wants to know "what niche is Dark really good at?" And unfortunately the answer to that is far less specific than I would like, as a lot of tech is designed to let you do whatever you can come up with. "Programming language but for retail" makes about as much sense as "Kafka but for retail" or "computers but for retail".

We actually tried to do something like this with Slackbots, you may remember. We tried to get 300 Slackbots written in a month or so at one point, and I think we got around 15 written. The problem is that people just don't have that many Slackbots to write in a particular week or month. And when they do, they have the same problems that other people have, which is that the thing you want to connect your Slackbot to can be just about anything.

So we've struggled with positioning, and one of our big problems has been telling people what it is that we do.

Current State of Dark

Currently, I believe that Dark is a pretty effective MVP, but is actually further from product-market fit than I initially thought. The reason for this is that features that Dark needs to be successful are often hidden below either product or technical debt. We've gotten some very effective proof points that Dark is on the right track, in particular that the core things that I think make Dark great (Deployless, Trace-driven development, and the lack of infrastructure) are things that people love.

Roadmap

The connection between the MVP and the positioning might not be immediately obvious, so let me spell it out. Our positioning tells people that Dark is a particular thing, namely the easiest way to code backends. And then they try Dark and discover that that is not the case. And then they wander away.

I think both positioning and the product are missing the same thing, which is a sense of where Dark is going. To illustrate what I mean, here's an extremely rough draft of how I think I should position Dark:

Dark is trying to be the easiest way to build an API. It's experimental, shiny, with interesting features, and very very incomplete. Here's a list of what's interesting. Here's a list of what you can do with it. And here's a list of what sucks about it.

Then the website would, for each feature/use-case discussed, include the relevant parts of the roadmap right there. For example "Dark is currently good for teams of two or fewer. To handle teams of 10, we'll need Operational Transforms, global feature flags, a history of changes made, and code review".

Not coincidentally, this same problem (the lack of where we are going) has come up in the two contributor meetups we've had. People are eager to help, but unclear of how exactly to do so, beyond the fairly chore-like issues in the issue tracker. So a roadmap should help there too. Unsurprisingly, this was also a problem for the engineering team before.

Next steps

So I'm going to try to produce a roadmap over the next few months. I think a good way to approach this is to try to write some app in DarkV2, without actually implementing any of it, just essentially writing out the code in a doc. Then, as I "use" the DarkV2, write the spec explaining what exactly these features are, how they work, etc, focusing especially on getting enough detail to outline the difficult bits.

The aim is to say where Dark is going, so that we can say how we are going to get there, and present that in a useful form for users, contributors, etc.

The difficult bits here refers specifically to the things that we've learned in Dark so far. Like, how the HTTP "framework" is inflexible, or that it sucks that you can't write your own types, or that Dark doesn't have GraphQL built in, or that we're relying on traces in places where a type-system would be much more useful.

And in the meantime, I'm going to be doing a bunch of learning about various technologies (eg GraphQL, Rust) - as I'll be replacing large chunks of the product, now is as good a time as any to think about their implementation.

You can sign up for Dark here, and check out our progress in these features in our contributor Slack or by watching our GitHub repo. Comment here or on Twitter.

A fun bug

Paul Biggar — Tue, 04 Aug 2020 16:15:27 +0000

I'm always interested in stories about Accidental Complexity—when you waste time dealing with the endless layers of tooling that's required to make a web application today. I'm also super into using Observability (which I'll call to ability to introspect live systems, though that's not exactly right). Fortunately for these two hobbies, I caused a partial outage of one of our systems due to terrible Accidental Complexity, and solved it using some delightful Observability. It's Story Time, gather round!

The outage

There were two indicators that something was wrong. Our "presence" feature, the thing that indicates that two or more users are working on the same canvas or handler, did not appear in a few instances. This isn't a commonly used feature as most of our users are working on small projects where multiple users aren't collaborating at once. It also breaks occasionally and fixes itself, so I ignored it for now.

The second was a user saying that traces weren't appearing for workers. To send data to a background worker, you call the emit function with some data, and then the trace appears on that function and you use it to do some Trace-Driven Development. However, the traces weren't appearing.

Pushing

When everything is working, traces appear because the background worker sends a notification to the client via our websockets vendor, Pusher. I could see from the Pusher stats that we had way fewer success messages than normal: we usually push through 1M messages a day, but for the last few days we had only been pushing through about 80k messages.

We had a log when Pusher messages fail, so I went to look at that. No, that's not true. First I sent a polite but distraught email to the fine folks at Pusher, asking what was wrong. They replied, and this is almost a quote: "Have you tried looking at the status code of the failing HTTP calls?". Oh right, yeah.

OK, so when I looked into that, there were indeed logs. We send all our logs to Honeycomb, which is a wonderful tool for ~~solving outages~~ debugging live systems.

This is the number of failing messages and successful messages. The vertical marker lines show deploys. Pretty obvious that a deploy caused this problem.

Looking at the deploy though, I have no idea what went wrong. Have a look at the diff and see if you can spot it.

Not only could I not spot the bug, but I couldn't replicate in my local dev environment either. Fortunately, Honeycomb came to the rescue.

Honeycomb has this great feature called "Bubble Up". Basically, you select a region of data on the graph, and Honeycomb tells you what's interesting about it.

Looking at this graph, we can see that some Kubernetes pods do not cause errors, and some pods sometimes cause errors and sometimes don't (no pods just issued errors).

Digging in further we see that errors occur far more frequently in the Kubernetes pod named "qw-worker" - the queue worker.

This makes sense, the background worker is where the problem appeared, so seeing that the problem is from the queue worker Kubernetes pods helps narrow it down. (Obviously, the queue worker system has many moving parts, not just that Kubernetes pod).

But, knowing the queue worker Kubernetes pod is responsible, it's easy to find the mistake. Spot the difference:

I had read this piece of code 20 times, and my eyes just glazed over it. But Honeycomb found it easily! The fix was pretty easy once it was found:

And once that fix deployed, here's what happened the errors:

Discussion

The Accidental Complexity I discussed at the start was Kubernetes. The fact that setting a variable was 5 lines of code, duplicated across three files, in a format that allows easy mistakes like this, is the root cause.

Even worse was the solution: I added two linters, yamllint and kubeval to try and prevent this from happening again (thanks to Dominic Evans for the suggestion. This involved writing this script, renaming a few dozen files, and reformatting a few dozen files as well. Of course, along the way I needed to learn how kubeval worked to see if the false positives were real and what to do about them.

Each part of this was Accidental Complexity. The search for the bug took a day, the simple fix took five minutes, and making sure this didn't happen again took a day and a half. No wonder everyone thinks backend is hard.

Another thing that's clear is how valuable observability is. Having the ability to search across live data was crucial here. One thing that is annoying is that we had to annotate this log data ourselves, and if we didn't have it we'd have been in trouble. Dark's version of this automatically logs, though it doesn't have good retention or graphing or the ability to identify trends--something to think about in the future.

Finally, it's clear that alerting on errors is important. We had logs being sent of the errors, but we did not have alerts about a new source of errors. As a result, the problem was happening for several days before it was reported.

Symptoms

Relating the cause and solution back to the original issues is also interesting. I observed that "presence" collaboration features weren't working. That's because they're built in Dark, and they use the worker queues - when those queues are backed up, they don't deliver. Since they're low priority this isn't a big deal, but something to think about as we do more Dark-in-Dark.

You can sign up for Dark here, and check out our progress in these features in our contributor Slack or by watching our GitHub repo. Comment here or on Twitter.

Evolving Dark's tracing system

Paul Biggar — Tue, 28 Jul 2020 15:52:37 +0000

One of the things that makes Dark truly unique is what we call "Trace-driven development". The best way to write a HTTP handler in Dark is to start by making a request to the non-existent handler, then:

using the 404s list to create the handler

using the actual trace value to see the output of your code as you type it.

We use the trace system a lot, and it's pretty great. It acts as a sort of omniscient debugger: you don't need to start it, you can go back in time easily, you don't need print statements. You can even see the control-flow of your application.

Like most things in Dark today, the trace system was built using the simplest, most obvious implementation possible. As we've grown quite considerably since then, we need to ensure that traces continue to scale well, which they currently do not.

This post is discoveries about what's not working, and ramblings about what the next gen should be.

Cleanup

Dark stores basically every request that is made to it. And it stores it in the database. While this data is important, it isn't the same level of importance as user data. Storing useful and volumous data in the same DB as a much lower volume of extremely precious data is not a great idea.

To avoid the DB blowing up in size (and price) we go through the DB and garbage collect it pretty much continuously. We keep the last 10 requests, and also keep any requests that were made in the last week.

We have struggled to make this not be incredibly buggy. The logic is tricky, and mostly written in SQL whose performance is iffy and which hides quite a few footguns. As a result, the requests to delete data are slow (this garbage collector provides the majority of the load on our database, interestingly) and also locks quite a bit (though I'm systematically working through this in a recent PR).

It can also be hard to identify what data to delete. When we started, we didn't know how we wanted traces to work, and so went with an implementation that stored a trace using the path of the URL requested. This worked well initially, especially as it allowed for easily transitioning a 404 (essentially, a trace with no owner) to a new handler, but had weird behaviour when you changed a handler's route (losing all its traces!). Alas, URLs also support wildcards, and so this meant that in order to find out whether a trace should be deleted, we basically had to recreate the entire routing business logic in the DB.

My thinking here is to associate the trace with the actual handler it hits. That way we're not recreating the business logic, but we'd need a separate 404 storage (although this is probably simpler in the long run). It also changes the behaviour when you "rename" a handler, which you sometimes do early in development; the new behaviour would be to keep the existing traces, which honestly is a much more user-friendly behaviour.

Storage

One of the problems is that we're storing the data in a DB. This sort of log data, which is mostly immutable, should be stored somewhere more appropriate, like S3 (we use Google Cloud, so Cloud Storage in our case). This was also a pattern from the early days of CircleCI - we started by saving build logs in the DB, before moving them to S3.

That would also allow us to send traces to the client without going through the server, which has operational problems of its own. This solves a big problem for customers with larger traces, which can time out when loading from our server. Since Dark is basically unusable without traces (you cant use autocomplete well without them, for instance), solving this is pretty important.

The other upside of this is that rather than running a GC process to clear up the DB (which doesn't even do a great job, as the DB will continue to hold onto the space), using something like S3 would allow us to have lifecycle policies to automatically clean up this data.

One of the problems here is that traces aren't quite immutable. You can -- by intention -- change the contents of a trace. While the initial input is immutable, you can re-run a handler using the same inputs, which currently overwrites the same trace (users have found this dumb, so losing this behaviour is probably an improvement).

You can also run a function you just wrote, adding it to the trace. This behaviour actually is good - it's a key part of Trace-driven development that you start with a partial trace based on your inputs, and then start to build it up as you write code.

My current thinking is to add the concept of a trace "patch". If you run something on top of the trace, we store the "patch" in the DB and resolve/combine the "base trace" and its patches in the client.

Expiration

The GC process isn't a great feature. While it would be much better if it didn't hit the DB at all, it would be even better if it didn't exist. Cloud Storage/S3 have expiration policies, which can automatically delete data without having to go through an expensive GC process.

One issue would be that we don't want the latest ten traces (or some number) to expire. I haven't fully thought this one through, but it seems doable.

You can sign up for Dark here, and check out our progress in these features in our contributor Slack or by watching our GitHub repo. Comment here or on Twitter.

First Dark contributor meetup

Paul Biggar — Tue, 21 Jul 2020 12:40:48 +0000

On Friday we had our first contributor meetup. Thirteen folks showed up to a zoom call to discuss how to contribute to Dark.

Obviously, the point of the conversation was to get to know each other. We're all hanging out in the #contributors channel, but the room isn't super chatty so talking in person (or whatever counts for in-person on Zoom) is helpful to get to know one another.

Big projects

One of the big topics was how to do bigger projects. Most folks had fixed some small bugs or added some tests, and were quite interested to contribute bigger features that would solve problems that were annoying them.

GraphQL

The first of these was GraphQL. Folks wanted to be able to easily create GraphQL backends in Dark, which is obviously a great idea. I hope that folks will be able to create Dark backends using any protocol that makes sense to them, like GraphQL, gRPC, Thrift, etc, in addition to the REST APIs they can build now.

So what's the blocker here? Well, there's probably two. First, we need to design this; we need to figure out what GraphQL means in the Dark world. My instinct is that it's tied to types (which don't properly exist yet), and to DBs (which should use types but use schemas instead right now). But I'm not expert in GraphQL, and have never really used it beyond "hello world" apps, which means that making a really good GraphQL design for Dark is something that someone else would need to do.

Secondly, there is a lot on my plate right now, and GraphQL isn't near the top. So for this to happen, someone else needs to drive it. I certainly have the bandwidth to help someone drive this if they're interested (and a few people in the call certainly were, both for GraphQL and the other things I mention here), but I only have so much space in my head for things I'm personally driving.

I think this basically counts for most things in Dark: if someone steps forward to design something and drive it forward, there's a lot of latitude to do so, and I'm willing to put in the time to help.

Language features

We also discussed expanding the language. There's quite a few language features in the issue tracker already, and there's probably a need to add more. Dark needs tuples, enums, a type checker, characters, statements, refs, polymorphic types, more patterns to match, and string interpolation. We also currently have our Dictionaries and Records as a single concept, which is obviously (and predictably) bad.

This brings up an interesting question: how to trade off people's interest in contributing with the existing Dark vision. The Dark language is pretty much designed, though it is far from complete.

While that's not true for other parts of the platform, it is the case that there is a high-level vision for Dark (which is not necessarily well-articulated) which each new feature needs to fit into. Working with contributors to design new features will likely help expose that vision, so that's a great place to spend time.

Datastores

The final area we discussed contributions was around Datastores: folks wanted to add default values for schemas, and build DB migrations, both of which are very valuable. They are blocked by an awkward step though: Datastores have schemas, but they should be specified using types instead.

Contributing with other skillsets

Something else that came up was about contributing if you don't want to write OCaml/ReasonML. While there are existing docs about contributing in other languages or other ways, folks were also interested in helping spread Dark by writing about it, helping it spread, and other kinds of "content marketing". All of which is very welcome!

Next contributor meeting

Folks seemed excited to meet every 2-3 weeks, though people asked to move it a bit later (7am is early for US west coast folks) and to do it on a weekend. The weekend-vs-weekday thing is a bit tricky: we want to be inclusive to folks who are using Dark for work and don't have time on weekends, but also to people who aren't using Dark for work and can't take time during the workday. So we'll maybe try alternating. The next one is Saturday, August 8th.

You can sign up for Dark here, and check out our progress in these features in our contributor Slack or by watching our GitHub repo. Comment here or on Twitter.

Dark devlog #3: onboarding

Paul Biggar — Mon, 13 Jul 2020 22:54:47 +0000

An annoying thing about coding is that some days you're on fire, producing masterpieces, knocking down obstacle after obstacle. And some weeks, you're trapped figuring out bugs that ultimately come down to having a semi-colon out of place.

Most of the time for me, the difference between the two is the amount of context I have in the codebase, and the amount of technical debt that has been built up.

Onboarding

So this week I've been working on onboarding, and it's been a time of building up context on a repo I'm not familiar with, as I wade through technical debt.

The signup flow for Dark was interesting. You would sign up by adding your email address via a form on darklang.com. That form is served by the ops-corpsite canvas, and when you add your email you get added to a Mailchimp mailing list, and you get sent to one of two pages: if you have a signup token, you go to /signup, otherwise you go to /beta.

The beta workflow is the one we've used the most: it asks you questions intended to figure out how serious you were, so that we could do some triaging and customer interviews to make sure we understood what people used Dark for. This was helpful when we were in the early days. Potential beta users were added to an Airtable where we'd figure out the best users and email them via a form built using the ops-adduser Dark canvas.

Are you lost yet? We've barely started.

Users who replied with their desired username got added as users, which meant we'd fill out a form pointing to a little app we built in Dark. We'd auth with our personal gmails so that users would get a person email from our gmail when they were added. The app would also add them to the Slack community, move them to a different column in Airtable, send a handful of emails (via Mandrill via Mailchimp), send a Slack message to our Slack, and also a handful of things like tracking in Heap/Segment.

All built in Dark, including homegrown error tracking tool in Dark as well.

Of course, that's only one path. If you got sent to /signup you'd get to actually sign up. We'd check the token you were using was valid, and that the username wasn't taken (we expose some of the internals of Dark to certain admin canvases so that we can build non-essential tooling in Dark, such as this). You'd then be added to the app, and we'd inform Auth0 that we'd done so. Then you'd get an email telling you go to back to darklang.com, where you'd click Login, then click Login again, then click "Forgot password", then wait for an email and click on the link, then change your password, and then hopefully log in.

Not a great experience.

Of course, after that you'd be presented with a blank screen and have no idea what to do, and invariably you'd wander off. In the last few months we added tutorials and samples, so that at least you have some idea what to do, even though they're not great yet.

So I spent most of this week understanding and detangling this flow, which spans three dark canvases, 3 different codebases, and at least 6 different services/vendors. When you have 3-4 people involved in managing a user sign up flow, the complexity builds up, and while those complexities provide value, that value doesn't hold when there's just one of me. So I removed the vast majority of this, and made it so you could just sign up. That shipped on Friday, and it seems like 20-ish people have signed up since then.

I haven't announced it yet because the onboarding experience isn't good yet (you still, for example, have to go through the "forgot password" flow), and I still have to add analytics to understand what's actually happening (I don't know how many of the 20 got further, for example).

I've also spent time making it easier to learn about Dark, mostly by pointing everything at everything else; for example, you can now find most of our content from the footer of either darklang.com or darklang.com/docs. We have a wealth of video tutorials and doc tutorial and samples, which were not really linked from anywhere, and I plan to add those to within the app as well.

So slow but steady progress on onboarding.

Contributors

The contribution front is going pretty well. Most of the Changelog this week comes from contributors, with stellar work from Brian Chen and David Cooley.

David Cooley also made an excellent point about how people can contribute - if right now they have to learn OCaml they're not going to contribute as much as they would if they were able to contribute in Dark.

We've got the first inaugural Dark contributor meetup this Friday (you're welcome to join, even -- or especially -- if you haven't contributed yet). It's via zoom of course, as we're not monsters, and so far about 16 people are coming.

My hope is to figure out how we can enable more Dark users to contribute, so a lot of the conversation will be centered around things like the package manager and enabling contributions that can be written in Dark.

You can sign up for Dark here, and check out our progress in these features in our contributor Slack or by watching our GitHub repo. Comment here or on Twitter.

Dark devlog #2: what next?

Paul Biggar — Mon, 06 Jul 2020 14:30:24 +0000

After spending the last week on some extremely urgent things on my plate (helping the team get jobs, moving, infra stability, costs, and legal and transition work), I have time to get back to the product.

There's still not a clear sense of what to work on next. The positioning that we end up with is going to dictate the important work. However, it's clear that the main areas that need love in the near term, regardless of exactly what direction we're taking with the product, are:

positioning
contributors
onboarding
package manager

Positioning and long term

Positioning is what I talked about last week, figuring out what Dark can be that's useful in the short term (while sticking with the same long term vision). The main thing I'm zeroing in on here is trying to position around what Dark is really good at right now, while trying to maintain a reference of what Dark will be in the future. This would hopefully allow room to position as for example a "better zapier" while also allowing me to talk about building React apps, backends, internal tools, etc.

An alternative--or possibly a compliment--to the "better zapier" positioning that occurred to me is to create a forms library. It's still pretty difficult to put a form on the internet and do whatever you want with it, and you can see for example Netlify adding products that move in that direction. So what if we made a Dark library that had an experience as good as Typeform, backed by Dark, so that you could create a form with a super customized backend. This would allow fully-custom actions on each event, rather than just getting the results in a spreadsheet (as an example, you could automatically send a slack message when someone picks the "urgent" field from the form).

Contributors

After the initial excitement in the community about contribution, things have settled down to a core 5-10 people who are the most excited, which seems about right. I need to keep working on this, but I need to recognize that this is a journey. I'm going to set aside 30 minutes a day to work on contributor docs and tooling, to make sure that there's steady progress in making this better (this is on top of reviewing and talking to contributors).

I also need to announce that the Dark repo is public, and a blog post to discuss that. I'm planning to have a contributor virtual meetup as well.

One area that is needed is enabling contributors to contribute to build new features which use Dark as major components. While language and execution features need to be written in OCaml, most other parts of the product could be safely written in Dark.

Well, theoretically. Right now, permissions in Dark are extremely coarse-grained. There's no way to add a contributor to a Dark canvas used for customers, without giving them access to the entire organization we use for this, including customer data. This is obviously a core product problem in Dark as well, so solving this would be beneficial to devs using Dark as well.

Another area I'm going to work on here is refactoring the client/frontend; it's a bit of a mess and I want to separate out components, document types, etc, to make it easier to contribute to. 90% of the work to be done is on the client, so I think this will help a lot.

Onboarding

This is the place I'm going to spend the next while doing a deep dive into. Sydney and Victoria recently made great progress here, especially with a tooltip-based tutorial framework that I can work with.

I need to take a look at what the analytics are like so I can see a complete funnel from homepage to completed tutorial.

I'm thinking of having a tutorial menu, so you can be stepped through the tutorials in the docs without having a different page open.

While we've been making fairly broad product changes over the last year, solving things in the smallest way possible, I want to take a different approach to improving the product now. Instead, I want to make an amazing experience for a particular aspect of the product. So for onboarding, this is an opportunity to address core product problems, like the omnibox being weird, or layout sucking. Dark's positioning could also get dragged in here as well, as that's obviously the top of the funnel.

My first steps are to instrument the website, frontend, and client, so I have a baseline to measure from. I saw Wisdom on HN recently and this seems like an amazing tool (especially given our user experience information is currently split between Rollbar, Fullstory, Segment and Heap). I'll try instrumenting with this and see what I learn about the funnel is like.

Package manager

Having a package manager is important, as Dark developers are pretty much just calling 3rd party services by manually writing HTTPClient calls. With all the different constraints I'm not really sure that I'm prioritizing correctly here. But you gotta pick something, and the package manager has more unknowns right now. I think this will be next after onboarding, though if contributors want to take it up, I could imagine spending time helping on it in the near future.

Any direction we take needs both the package manager and good onboarding, so this will get done either way.

Costs

The other thing that I'm keeping an eye on is costs. We're spending quite a bit of money on things like cloud services that we're not really using, and it seems like we're massively overpaying for capacity on Google Cloud, relative to what we're using. Technically it's free but we'll run out of credits eventually and it's better to be ahead of this, so I'm spending a small amount of time each day looking at this. Today I reduced the size of our read replicate, which is basically unused, saving probably $500 a month.

tl;dr

In the short term, I'll be spending a lot of time on onboarding, and a smaller amount of time on costs. I also plan to continually spend time on contributors, especially on docs.

You can sign up for Dark here, and check out our progress in these features in our contributor Slack or by watching our GitHub repo. Comment here or on Twitter.