DEV Community: AriFordsham

What I don’t like about Github Copilot

AriFordsham — Tue, 01 Jun 2021 00:00:00 +0000

DISCLAIMER: I haven't had a chance to use Copilot - I'm on the waitlist, like everyone else. I'm writing based on the impressions I've got from the quite nice website, the underlying OpenAI Codex technology, and my own rudimentary ideas of how machine learning works.

Microsoft, the owners of Github, annouced that a new project -- Github Copilot -- entered Technical Preview on the 29th June 2021. It's an extension to the Visual Studio Code editor that analyzes your code and suggests completions -- even multi-line functions -- based on a machine learning algorithm, that was trained on all of Github. From the online examples and testimonials, this sounds like a a Big Deal.

You are now invited to join the waitlist and who knows? you may be one of the lucky ones who gets to give it a spin. Spoiler: Github have said "we are offering GitHub Copilot to a limited number of testers for free" and "if the technical preview is successful, our plan is to build a commercial version of GitHub Copilot in the future." The production version almost certainly won't be available free, and probably won't be cheap. If it's as good as it promises, it could become a staple of commercial programming shops, but indie developers like me who don't need this might not be able to justify it. Oh well, just saying.

I think Copilot does have genuine promise, and the possibility of making meaningful improvements to the process of coding. But from what I've seen -- and this makes sense -- Copilot does not create. All it does is attempt to piece together an ad-hoc description of the programmer's meaning from information embedded in code, comments and names into hopefully correct executable code. Your informal description probably needs to contain barely less detail than the code itself. In this way, Copilot transforms and sharpens, rather than innovates¹.

There's a joke going around: after Copilot comes Pilot, and then I'll have to get a new job (if there are any). Jokes aside, getting from Copilot to Pilot is WAY bigger than getting from nothing to Copilot - it's not even the same kind of thing.

But here's the problem. The most fundamental principle of software development is DRY -- Don't Repeat Yourself. Repeating the same code in two places is a code smell - you should find some way of writing it only once, such as putting it in a function.
The formal name for avoiding repetition is abstraction.

Now, in order for a model like Copilot to learn, it needs a lot of redundancy in it's dataset. The success of Copilot hinges on the fact that in the code we write today, there is still loads of duplication -- we're not nearly there yet. And the real solution to redundancy is getting better at abstraction -- finding and using new abstraction mechanisms, and learning how to apply abstraction more widely and effectively².

Copilot's breakthrough party trick is the ability to analyze natural language in comments and function and variable names to come up with suggestions. However, many of the examples on the website show that Copilot can -- and does -- suggesti multi-line code snippets, the kind of thing I look at and think: that shouldn't be pasted in, it should be abstracted away. By doing this, Copilot is locking in a duplicative coding style. At the same time as Copilot is doing your work for you, it's also taking away the pain of being verbose and repetitive and thereby encouraging copy-paste-style coding. I worry how this could hold back progress on searching for abstractions. Abstraction is more than saving developer time -- it can even slow you down up front. Rather, it helps make code more readable and maintainable (by moving irrelevant detail away), it can even make code more correct and performant (by spending the time getting it right once), but above all -- it adds to our understanding of software development -- it lets us recognise these pattens, even give them a name, and thereby guides us to think higher level. Blindly pasting code from Copilot doesn't do any of these things.

Of course, on the flip side, Copilot might encourage developers to write comments, which is a good thing. Then again, they probably won't delete or trim the comments after they write(?) the code, and therefore the comments will repeat what the code says, which is bad³.

Can we use AI to find abstractions? Very possibly, and that would be amazing. but it will be a lot harder than merely analyzing and parroting code, and no one the size of Microsoft/Github is even trying.

I'm not knocking Co-pilot -- it definitely seems to be, rather than a gimmick, an amazing product that has the potential to transform the way we code, and especially help developers bridge the gap from junior to proficient. I just hope the activity of programming will not take a step back as a result.

This observation, by the way, can be generalized to every machine learning (ML) project I have seen to date -- very powerful but blunt generalization tools that don't really have more than 'one level' of insight (whatever that means.) Can AI do more? Very possibly, but I haven't yet seen a single proof-of-concept. ↩
At this point I'll give a plug for my favorite technologies: pure functional programming languages, especially Haskell. Functional languages take abstraction very seriously -- they have a bunch of powerful abstraction mechanisms, and their communities try very hard to discover new ways of encoding common patterns. Haskell programs use fewer lines of code to accomplish the same tasks. It would be very interesting to me to see if Copilot is less effective on Haskell because it has less redundancy. ↩
See Chapter 1 of The Practice of Programming for why. ↩

How I reduced my Haskell CI time by 84%

AriFordsham — Fri, 14 May 2021 00:00:00 +0000

I know how confounded I was by CI before I got into it, and how straightforward it seems now, so I thought I'd write down my experiences for anyone following in my footsteps.

In this post, I will speak about how I implemented Gitlab Continuous integration (CI), and then sped up execution time dramatically by setting the right options. My project is Haskell-centric, but many of the takeaways can be applied to any language.

I use Michael Snoyman's excellent stack build tool for my Haskell projects. Amongst many features that generally improve my quality of life as a developer, stack tries to guarantee reproducible builds - controlling as many variables as possible, to ensure build behaviour is the same - reproducible - between builds, even if I change the configuration of my machine, or even move the project to a different machine.

To achieve this, stack downloads all project dependencies into the project directory. But it goes further - it downloads a fixed version of GHC, the Haskell compiler, and stashes it away in a special location. This ensures every version uses a specific version of the build toolchain, and updating my system GHC install will never break a project due to a subtle change in compiler behaviour, for example.

I have a shell script I use that defines what I consider a full working build for my project, stored at ci/build.sh:

stack build --test --coverage --haddock $@

Run the stack build command to build the project (based on the poject's package.yaml file.)
Run all test suites
Generate a test coverage report
Generate haddock documentation
The $@ passes command-line options from the script through to stack build - useful for one-off build scenarios.

My project is hosted on Gitlab, so I wanted to get GitLab's excellent CI to run and validate this script on every push.

Initial setup was dead straightforward (once I knew how!): I created a .gitlab-ci.yml file in the root of my project as follows:

image: migamake/stack-build-image:17.0

stack:
  script:
    - ci/build.sh

Define which Docker image to use. I'm using stack-build-image from Migamake, which provides stack preinstalled, for LTS Haskell version 17.
Define the stack job, which runs the script ci/build.sh.

Job done! On every push, Gitlab runs the build script in a Docker container. Since stack returns an error code if any test suite fails, I get a big red cross next to my commit on GitLab if my project fails to build or run correctly.

Here comes the issue: after writing that script, ever CI run took in excess of 33 minutes. Not that this is really a problem in any way: My CI script also runs in a pre-commit hook¹ (which runs in under a minute on my development machine), so I never push without knowing CI passes. It's just nice to not have to wait long for that satisfying green tick, and with all those bitcoin miners around, It feels better not to be unneccesarily squandering the planet's resources.

```sh
sh ci/build.sh
```

On Linux, you'll generally need to set this as executable by running `chmod a+x .git/hooks/pre-commit`.

This will now run before every commit. As with CI, an error code from the script will cause the commit to fail. You can override this with `git commit --force`.

A quick look at the build log shows the problem. As mentioned, stack downloads the GHC compiler and all dependencies. Since Docker does not store any state between runs, the build system has loads of downloading and compiling to do before it can even start building and testing the project itself.

That's just unnecessary duplication of work - the dependencies don't change between runs. On my local machine, stack keeps a cache of already-built libraries, so rebuilds are near-instant. However, to ensure reproducibility, all Docker runs start with a clean slate (besides whatever is in the container itself,) so this cache is not kept.

Gitlab's caching feature comes to the rescue. You can tell Gitlab that changes to certain directories won't affect the build correctness of your project, so Gitlab will go ahead and preserve those between runs. I added the following to my .gitlab-ci.yml:

cache:
  key: "ALL"
  paths:
    - .stack-work/

This sets up Gitlab's cache for the .stack-work/ directory, where stack keeps library dependencies². Now, at the end of every run, Gitlab zips up the .stack-work directory, and uploads it to a cloud bucket. Then, next time, before it begins running your CI scripts, it downloads and unzips it into your project directory.

This should allow stack to now find its cache from last time, and should enable zippy-fast builds, like I get on my local machine.

However, pushing the build script (twice; once to create the initial cache, and then again to see how it's used) gives only a negligible improvement in CI run time. A glance at the build log seems to show that the cache is working correctly. So what is going on?

stack caches project-specific artifacts in .stack-work, under the project directory. However, the bulk of its local cache, including the GHC build chain and many libraries, is stored by default in .stack under the user home directory. This enables the sharing of the cache between projects.

Now there's an additional complication: While Docker containers provide a home directory for projects to use, Gitlab cache only works for directories under the project directory. So I also need to set the $STACK-ROOT environment variable, to tell stack to store its build chain where the cache can see it.

My final .gitlab-ci.yml looks like this:

image: migamake/stack-build-image:17.0

stack:
  script:
    - ci/build.sh

variables:
  STACK_ROOT: "$CI_PROJECT_DIR/.stack"

cache:
  key: "ALL"
  paths:
    - .stack-work/
    - .stack/

Success! After generating the cache, run time has dropped from 32 minutes to 7 minutes.

A look at the build log shows that this time is dominated by downloading and uploading the cache zip. Even adding --fast to ci/build.sh, which tells stack to do unoptimized builds, made negligble difference, so I concluded there's no more simple optimization opportunities.

Since I initially set up CI, my project has grown and added dependencies. So I ran the latest version, with and without the cache enabled.

A regular run now takes just under eight minutes, but without the cache, it takes almost 50 minutes, or an 84% reduction in run time.

CI looks daunting, but it simple enough once you invest the time to learn how to do what you want, andif you do it right (and don't overcomplicate!) you can significantly improve and smooth your workflow.

Here's how you do a precommit hook: ↩

Your git repo has a hidden directory, .git, where git stores it's bookkeeping. Under that directory is a directory called hooks (By default, it contains a set of .sample files showing what hooks are available). I created a file there called pre-commit:
The key field is required: it allows you to use different caches for different scenarios, such as branches, by setting the key to an environment variable. I trust stack to always use the right library versions from the package.yaml, and so the more sharing the better. Therefore I just use an arbitrary contant string so all scenarios share a single cache. ↩

Haskell - Doomed to Succeed?

AriFordsham — Wed, 16 Dec 2020 00:00:00 +0000

The unofficial motto of Haskell, the predominant lazy functional language, has long been:

"Avoid success at all costs."

This is attributed to Simon L. Peyton-Jones (SPJ), the main architect of the Haskell language and the GHC compiler, and mentioned in the slides of a talk he gave at the POPL conference in 2003.

This seems an odd statement. What's wrong with a bit of success?

The exact meaning of this statement is controversial. Simon Marlow, another major Haskell figure, quotes SPJ that this statement (expression?) should not be bracketed (avoid success) at all costs, i.e. do all you can to ensure success doesn't happen, but rather avoid (success at all costs) - success may be a nice idea, but avoid falling in to the trap of focusing all the community's efforts to attaining it.

But isn't success the goal of any project?

The answer can be found in an article written by the aformentioned Marlow and Peyton-Jones:

[T]he ultimate goal for us, the main developers of GHC, is to produce research rather than code. We consider developing GHC to be an essential prerequisite: the artifacts of research are fed back into GHC, so that GHC can then be used as the basis for further research that builds on these previous ideas.

In other words, the success referred to is commercial success, or wide user adoption. The stated goal of GHC (and the Haskell language) is to better understand the principles of functional programming, and the mathematical and logical principles that underly them.

A wide user base, using Haskell in important projects, poses a responsibility to the Haskell designers. Users have needs, and can push the project in directions irrelevant to the core research goals, as the authors go on to state:

[A] great deal of effort is put into ensuring that [GHC] can be relied on for production use. There has often been some tension between these two seemingly contradictory goals, but by and large we have found a path that is satisfactory both from the research and the production-use angles.

They certainly seem to have been successful: GHC is widely considered a robust and reliable compiler, and the GHC developers take the quality of their product very seriously, despite their very limited resources.

So while SPJ claims to have been saying "don't allow efforts to make the platform appealing to production users take away from Haskell's key principles," It definitely has been understood in practice to mean "Let's try not to draw attention to ourselves, because too many users will prevent us doing what we want to do."

Until now.

In his POPL talk, SPJ quotes Anthony Hoare as saying:

"I fear that Haskell is doomed to succeed.”

Haskell's advantages - a strong mathematical basis, facilities for abstraction and de-duplication, excellent performance compared to other high-level languages, good correctness guarantees, ease of concurrency, but most of all, enabling a better way to think about code, have increasingly lead to its widespread adoption in production code. Haskell is contending with success despite itself.

This shift in the makeup of the Haskell community certainly has led to a shift in focus. At the Haskell eXchange conference in November 2020, SPJ announced the Haskell Foundation, explicitly "dedicated to broadening the adoption of Haskell" as well as "supporting its ecosystem of tools, libraries, education, and research."

So while the original architects of Haskell may find this unintended success unwanted, it is certainly a vindication of their vision: remaining faithful to strong mathematical principles and resisting calls for premature pragmatism can transform the things we do.

UPDATE

Simon Peyton-Jones has graciously commented on this post - you can read his remarks here.

In short, he reiterates that wide production adoption was never an anti-goal of Haskell, but rather not "making fundamental compromises of core principles [of Haskell] in pursuit of short-term production goals."”

Is Haskell fast?

AriFordsham — Tue, 24 Nov 2020 00:00:00 +0000

If you try to research this question, you might come up with confusing and contradictory answers, strongly-stated opinions, and loads of technical jargon.

The answer depends which perspective the questioner is coming from, and what assumptions, expectations and preconceived notions they bring with them.

Correctly written Haskell is fast for a high-level language

Haskell is a high-level language - a tool used to write programs with a goal of avoiding the programmer having to specify implementation details, as far as practical. This is typically the most productive way of writing most software, where performance must be ‘adequate’ rather than optimum.

In this space, developers typically reach for dynamically typed, interpreted languages such as JavaScript and Python. This kind of language design has an inherent overhead, because there is no opportunity to eliminate runtime work ahead-of-time, and in particular the program has to shuffle around and bookkeep type information.

Haskell, on the other hand, is type-erased and compiled. The GHC compiler does an excellent job of optimizing Haskell code to make it run much faster than it otherwise might. So as long as the developer is aware of the necessary caveats and pitfalls (more on those later), Haskell programs will run dramatically faster than their HLL counterparts.

Straightforward Haskell will be slower than C

It remains a fact that code written in clear, idiomatic Haskell will (at present) be slower than code written in a low-level language such as C, C++ or Rust, which would typically be used for performance-critical code. GHC cannot yet optimize all the overhead from the Haskell abstractions, and will lose out to code that runs close to the metal, typically with a slowdown of around 50% to 4x.

This leaves Haskell in a bit of a tight spot when it comes to performance: it competes well against other high level languages in a domain where performance isn’t that important, but can’t quite keep up with the incumbents in the race for ultimate speed.

However, if someone is using Haskell for other reasons, to write more concise, maintainable code that will be easier to be confident in its correctness, they will get decent performance and responsiveness ‘for free!’

You can write performance-competitive Haskell

So what to do with performance-critical code? One option is to write it in a low-level language, possibly calling in from Haskell through the Foreign Function Interface. In doing so, the programmer accepts the sacrifice of expressivity and safety, and must factor for the overhead of FFI and marshalling.

But there is another way. With a strong understanding of the GHC execution model, there is a set of techniques that can be used to write Haskell code that compiles to run at least as fast as equivalent C programs. This takes quite a bit of expertise to do, and it won’t be quite as tidy as idiomatic Haskell, but it should still be more readable (to the practiced eye) and maintainable than a highly-tuned C implementation.

A full rundown is beyond the scope of this post, but I’ll link to some resources:

Write Haskell as fast as C: exploiting strictness, laziness and recursion (Written in 2008)
All About Strictness Analysis (How to take advantage of the latest compiler optimizations)

Watch out for the Haskell performance traps

There are, however, some things that every production Haskell developer needs to be aware of. There is Haskell code that looks correct, but will run way slower, and use far more memory, than the worst you will come across in an interpreted language. The slowdown can run to several factors, with programs occasionally running out of stack space before they can get anything done!

This is due to a controversial design decision called laziness, compounded by, in hindsight, suboptimal library design and idiom choice, and needs to be understood in order to be able to guarantee consistent reasonable performance from your Haskell code.

I won’t go into all the details here, but in short, never use foldl (use foldl' instead), know how to use bang patterns, and read this article by Michael Snoyman:

All about strictness

Abstraction doesn’t have to cost; it can even pay

People seem to think there is inevitably a price to pay for writing high-level code. In fact the opposite is true. Theoretically speaking, high-level code is easier to optimize than low-level code. It simply provides more information to the compiler about the programmer’s intent.

As an illustration, let’s take the case of Rust. Rust is considered on par with C and C++ in terms of performance, despite being somewhat higher level. In spite of this, if you look through the Rust issues on Github, you’ll see many instances of the ‘I-slow’ tag, suggesting unexploited optimization opportunities. This suggests Rust is set to get faster yet.

How can this be? Surely everyone knows nothing beats hand-written C code?

I believe this to be a widely held myth. Optimization is more-or-less an exact science, and machines must eventually outperform humans. And ultimately, a perfect Haskell compiler will outperform a perfect optimizing C compiler.

Why is this?

A C program prescribes precisely how a problem is to be solved. There is a limit how far the compiler can go in rearranging the code without breaking the language semantics. A high-level program describes the problem, and gives the compiler more freedom to choose the best implementation.
Few programmers have a comprehensive knowledge of how best to optimize their programs. Compiler optimizations can be written once by a community of domain experts, and then applied widely.
Even the most experienced C programmer must keep objectives other than sheer speed in mind while writing his code. The program must be able to be evaluated for correctness without losing his train of thought, and it must be extendable and maintainable, often by other people. In the source code, these are unavoidable trade-offs.
Because of this, even programmers steeped in C performance technique must program defensively to ensure program correctness. A compiler can perform aggressive optimization that will make a programmer’s jaw drop, simply because it can keep track of far more program state at a time in order to gaurantee correctness. The compiler can then often build optimizations on other optimizations. Imagine trying to aggressively inline a program by hand.

Summary

As long as it your code is written with an eye to a few common pitfalls, Haskell is a fast language compared to it’s closest competitors, although not quite fast enough to compete in the speed stakes. Heavily hand-optimized Haskell can be competitive, and we can expect compiler advances to further narrow the gap for more idiomatic code in coming years.