LLMs as compilers

a brief history of our programming abstractions

It is hard to fathom, but in a not-so-distant past we considered assembly languages a high-level programming language when compared to its predecessor, machine code. In 1947, assembly languages introduced an abstraction layer to simplify programming computers by letting us use numbers, symbols, and abbreviations to talk to computers instead of just 0s and 1s as had been the case with machine code. Assembly languages gave birth to computer science as a field in the digital realm if we exclude analog technologies from the 1800s and early 1900s like mechanical calculators, abaci, tabulators, etc. However, assembly languages were still very low-level and required a lot of effort to write and maintain.

; A sample “hello world” program that writes to stdout using `write` then exits using `exit` on Linux x86-64.
; courtesy of Jim Fisher https://jameshfisher.com/2018/03/10/linux-assembly-hello-world/

global _start

section .text

_start:
  mov rax, 1        ; write(
  mov rdi, 1        ;   STDOUT_FILENO,
  mov rsi, msg      ;   "Hello, world!\n",
  mov rdx, msglen   ;   sizeof("Hello, world!\n")
  syscall           ; );

  mov rax, 60       ; exit(
  mov rdi, 0        ;   EXIT_SUCCESS
  syscall           ; );

section .rodata
  msg: db "Hello, world!", 10
  msglen: equ $ - msg
        `

Then came Fortran I in 1953, with a new I/O interface abstraction that allowed us to more eloquently ask computers to perform and display numerical calculations. Fortran I was the first general-purpose programming language of its kind, but it would later become the common ancestor for many more of these syntaxes including C in 1972. However as we started to write more complex software with Fortran I, we realized that manual memory management became more and more cumbersome. John McCarthy came up with Lisp in 1959. Lisp was the second general-purpose programming language to exist after Fortran I and provided a higher-level abstraction that automatically managed memory for us using a new concept at the time called garbage collection. Automatic memory management of a computer's memory wasn't (and still isn't) as performant or efficient as the manual allocation and deallocation of memory, but it provided us with a superior software development experience.

The evolution of programming languages has been a constant march towards abstractions that make programming easier and more accessible. Newer versions of Fortran and the majority of the most used programming languages today use some descendent of the original garbage collector invented by John McCarthy to automatically manage computer memory. Programming abstractions build on top of each other over time and the old ones continue to live underneath the newer ones that we use today. And so the high-level languages of yesterday become the build artifacts and implementation details of today.

how deterministic does a compiler need to be?

LLMs are already changing the way we interact with code as witnessed with the advent of GitHub's Copilot. However, thus far LLMs have only served as programming companions that we can't fully rely on as they are not deterministic and can periodically hallucinate or produce errors. Historically, compilers are deterministic. Given the same input, the same compiler will always give you the same output. This has ruled out LLMs as compilers in the traditional sense. However, how much determinism do we really need to reliably produce software that does what we want it to do? If we think of an LLM as a software engineer, as opposed to a companion, the same software engineer can produce different software given the same spec or set of requirements. All the outputs across different implementation attempts can be valid unless the requirements provided are excessively stringent. If we can get to a point where LLMs can reliably produce correct code that meets the requirements given, then we can start to think of LLMs as compilers even if they are not deterministic. The question is not really if it will happen, but when it will happen. The field of generative AI is unfolding rapidly and working towards both the correctness and determinism of the code LLMs produce. As we make progress on both fronts, LLMs will sooner or later add English as another root node of the genealogical tree of programming languages, and from it, a whole new set of syntaxes will span that will use the high-level programming languages of today, like Javascript and Python, as foundational and low-level implementation details akin to what assembly represents today.

compillmers

There is already a lot of hay to mow with the current state of affairs in generative AI. LLMs as proper compilers, compiLLMers if you will, can produce correct code reliably enough today given enough guidance. Getting an LLM to generate correct code requires providing various examples and descriptive instructions. The UX of a chat interface to an LLM inherently leads people to write prompts that do not meet these criteria. We need to make it easy for people to give LLMs precise descriptions and numerous examples as concisely as possible via syntaxes that are similar to English so they remain easy to learn and use. Coq is a great example of a functional programming syntax that is verbose and distant from English, but example-driven via assertions. David Ellis, Alejandro Guillen and I recently introduced Marsha as a proposal for what a syntax that meets the requirements outlined can look like. It is still early, but LLMs will increasingly give us the power to create more accessible representations of computer programs that look close to English. These representations will be distilled by LLMs into the complexities of the current high-level languages. Knowing Java or Python will become a rare skill, akin to individuals specializing in low-level optimizations using C or assembly language these days. Instead, the focus of developer experience will shift to the higher-level abstractions that are built on top of LLMs and composing these abstractions for different tasks. Compillmers will make programming more accessible in the near future such that writing software becomes part of the resume of most knowledge workers.