Pure Forth

Posted on Aug 1 Originally published at forth203x.hashnode.dev

Changes to the Standard, or, How Europe's Bush-Leaguers Murder Forth

#forth #programming #languages #opensource

Programming language success is determined not only by the language itself, but also by its ecosystem—the availability of good IDEs, debuggers, libraries, and other development tools. Against this backdrop, Forth has been steadily losing ground, even in the embedded domain that was once its natural home.

In my view, the problem is not just the technological revolution. It is also the gradual degradation of the Forth standard. Since 2012, the standards committee has made no meaningful progress in improving the language architecture, making Forth less attractive to potential programmers.

Under the pseudonym AbstractionFirst, I submitted a number of proposals to the standards committee. They did not even understand what the discussion was really about. Let's examine the issue step by step.

Today, Forth looks as if its guts are hanging out—its internal implementation mechanisms are exposed where a proper language interface should present well-designed abstractions.

First, the word BASE.

The API of BASE is poorly designed: it is exposed as the address of a variable. Why does the standard still expose the numeric base as the address of a mutable memory cell, instead of standardizing operations for reading and modifying this property?

The standard has standardized the implementation rather than the contract. What would the ideal design look like? It should follow a well-known API design principle: specify observable behavior, not the internal representation of state.

Every standard word should describe either the observable behavior of the language or a fundamental semantic capability—not the specific organization of an implementation's internal state.

At this point, there is no need to reduce the discussion to the question of what word should replace BASE. The more important point is this: the standard should not expose the representation of a language property as the address of a variable. It should standardize the operations on that property while leaving their implementation entirely up to the system.

I do have concrete proposals as well. One possible approach would be to use VALUE and TO.

The first person to respond to my proposal on forth-standard.org was a user under the pseudonym ruv. At first, he seemed to make a sensible suggestion by proposing the words RADIX and SET-RADIX. However, he concluded his comment with the rather strange remark that generating an exception through TO would be difficult. I found that statement quite puzzling.

In the context of BASE, where one numeric value is simply replaced with another, the idea of trapping an exception seems out of place. It gives the impression that ruv either had not fully thought through his response or was arguing merely for the sake of argument.

What a program actually needs is access to the current numeric base—not to the address of the memory cell where that value happens to be stored. This is the key point. In other words, VALUE was merely an example of a possible direction for the language to evolve, not the subject of the discussion itself. In fact, my proposal to the committee was about the architecture of the language rather than the name of a particular word.

The next response came from the committee chairman, Anton Ertl. His comment clearly carried an irritated tone. Let me say right away that, in my opinion, this is unprofessional. Fine. If Chairman Ertl chooses to engage on an emotional level, then I will respond in kind. All of my comments submitted to the committee were restrained and focused on technical substance. This article, however, is written at the level Ertl seems to understand better.

Incidentally, just like ruv, Ertl also failed to understand what the discussion was really about. My proposal was not merely about implementation mechanisms—it was about the proper architecture of Forth as a whole. But even if we focus on the details, I find his argument that "the proponents have not demonstrated why this is actually a problem" rather strange. It immediately raises a question: has Anton Ertl ever written a single line of assembly language himself, or does he merely supervise the Gforth project?

Today, the word BASE is never used by itself. It always appears as either BASE ! or BASE @. In threaded code, that means two operations instead of one. In other words, using BASE is twice as slow and produces twice as much code. As the author of my own Forth implementation, I know this very well. BASE is not only a user variable; it is also used by >NUMBER, which in turn is part of the Forth engine itself. In other words, BASE has become a performance bottleneck in the language.

Instead of introducing VALUE, another possibility would be to merge the pairs BASE ! and BASE @ into single words: BASE! and BASE@. Code portability would not suffer significantly—provided there is a reasonable willingness to evolve. Better yet, the standard should embrace the following principle:

“Backward compatibility is not an absolute value if it stands in the way of architectural improvement”.

Arguments that replacing one word (BASE) with two (BASE! and BASE@) somehow makes the language worse are not, in my opinion, a serious basis for discussion. The current standard already contains words that could reasonably be declared obsolete, just as was once done with TIB, #TIB, QUERY, EXPECT, SPAN, and CONVERT.

Some words do not necessarily need to be removed from the standard. It is enough to move them into a secondary role. And that brings us naturally to the second part of the discussion.

Second, the words CMOVE> and CMOVE.

Neither of them has any right to be in the STRING word set. Both words are inherently unsafe. There is already a reliable alternative—MOVE—that does not suffer from these shortcomings.

The only place where CMOVE> and CMOVE can be justified is in embedded systems, where every byte matters. Therefore, I propose that the next revision of the standard introduce a separate EMBEDDED word set and move into it all words whose existence is justified solely by the need for compact implementations.

That said, given the amount of memory available in modern microcontrollers, even the need for a dedicated EMBEDDED word set remains an open question.

Third, the words SAVE-INPUT and RESTORE-INPUT from the CORE EXT word set.

The standard has conflated two different layers of Forth: one intended for implementers and one intended for users. Users should be exposed only to abstractions that are natural within their problem domain. For most Forth programmers, that domain consists of the stack, the dictionary, numbers, strings, and files—not the internal state of the text interpreter.

This raises an obvious question: why are SAVE-INPUT and RESTORE-INPUT placed in the same namespace as ordinary application words?

Let me repeat: the standard has mixed together two different interfaces:

the API for writing application programs;
the API for implementing Forth itself and tools tightly integrated with the interpreter.

This reveals a troubling symptom. A language standard is normally expected to standardize observable behavior rather than implementation mechanisms. In this case, however, standardizing such words begins to dictate the architecture of the implementation, limiting the design freedom of Forth system implementers. If we follow this principle consistently, the standard should avoid prescribing internal implementation mechanisms even within the mandatory CORE word set.

Let me explain this in more detail for those who may not yet see the distinction.

There are language users, and there are compiler and system implementers. These are different roles. Historically, Forth has often combined them, but that alone does not imply that every internal mechanism should be standardized.

Language users need words such as EVALUATE, POSTPONE, IMMEDIATE, and INCLUDE-FILE. The existence of SAVE-INPUT and RESTORE-INPUT, on the other hand, reflects one particular model of how the interpreter is organized internally.

Let me be more precise: not every implementation-related word imposes an implementation strategy. For example, POSTPONE has a profound influence on compiler design, yet it expresses language semantics. Without it, a certain class of portable programs simply cannot be expressed. SAVE-INPUT, however, does not introduce new language semantics. Instead, it describes one particular way of organizing the interpreter's internal state. That is the fundamental difference.

I would summarize the principle as follows:

"If a word exists primarily to support a particular implementation architecture rather than to express the portable semantics of application programs or language extension tools, then standardizing that word begins to restrict implementation freedom".

The same idea can be expressed as a simple question:

"If a mechanism exists primarily for the implementer, why is it standardized as part of the language's user interface?"

Once again, the first response on forth-standard.org came from ruv. His reply makes it clear that he completely missed the point of my questions. My concern was never the quality of the API itself—it was the level of abstraction. His response barely addressed the architectural question I had raised and therefore missed the central point entirely.

Anton Ertl replied once again, and once again he failed to grasp the essence of the questions I was raising. I was talking about the form of the public interface, while he was talking about the many possible internal implementation algorithms.

I also cannot help pointing out once again that the committee chairman's response fell short of professional standards. If you read his reply carefully, you will notice how dismissively he spoke about certain Forth implementations, effectively placing them into a category of useless systems. In my view, that was a remarkably arrogant remark toward Forth implementers who invested their time and effort in creating their implementations.

Looking ahead for a moment, if we set aside the emotional aspect of the discussion, I am left with the impression that my opponents on forth-standard.org simply do not possess the conceptual vocabulary needed to discuss the questions I was raising. As a result, they never really understood what I meant.

Later, Ertl posted another reply, but once again the subject of the discussion shifted. I never asked whether SAVE-INPUT was well designed or well implemented. My question was different: by what principles should we decide which words belong in the standard interface at all? That is a question about the architecture of the standard—not about the quality of individual words.

The weakest part of his argument is the absence of any general principle. Instead, he evaluates words one by one:

Is it convenient?
Is it efficient?
Is it portable?
Is it easy to implement?

But he never answers the more fundamental question: "Should this word be part of the public interface of the standard in the first place?" I asked that question repeatedly in several consecutive comments on forth-standard.org.

I keep catching myself thinking the same thing over and over again: I am trying to discuss the rule, while most of the replies are busy discussing exceptions to the rule. I say, "Let's first establish the principle." They reply, "Well, SAVE-INPUT can be used like this..." Honestly, it leaves me with the feeling that I am talking to children. The committee members seem unable to think at the level of language architecture. It is simply not the level at which they approach these questions.

A little later, another user, pda, joined the discussion. His response was yet another childish argument based on conflating two distinct concepts. My response is straightforward. Forth has never prohibited programmers from working at a low level. But the language standard is a completely different matter. Its purpose is to define what should be portable. These are two different planes of discussion. The philosophy of "hide nothing" belongs to the capabilities of the language. It does not determine the obligations of the language standard.

In reality, SAVE-INPUT and RESTORE-INPUT are only the beginning of a much larger discussion—one about introducing a new word set into the standard. And that naturally brings us to the fourth and final part of this article.

Fourth, words such as >IN, SOURCE, REFILL, as well as SAVE-INPUT and RESTORE-INPUT, should be moved into a separate word set intended exclusively for implementers. Possible names include INTERPRETER, IMPLEMENTATION, SYSTEM, or ENGINE.

The rationale can be stated as follows:

"Words intended primarily to support the implementation of the text interpreter should be separated into their own word set rather than being part of the interface for portable application programs."

This is a question of the structure of the standard. It does not change the semantics of those words in any way.

The second architectural idea can be stated as follows:
"Even if such words remain standardized, their specification should define only their observable behavior rather than forcing implementers toward a particular internal model."

And this principle is not limited to SAVE-INPUT. It applies to any word in the language.

Anton Ertl responded almost immediately to my comment. In my opinion, at that point he crossed the line into disingenuous argument by claiming that the words I had listed are used by application programs. Let him produce a single convincing example. I would genuinely be interested to see what kind of extraordinary application he has in mind. As someone who has developed Forth systems and has also written Forth applications, I find it difficult to imagine why an application program would need to manipulate the Forth input stream directly.

Then ruv swooped in, declared that the issues being raised were off-topic, and closed the discussion. As far as I'm concerned, that tells me everything I need to know about them. They are incapable of engaging in a reasoned discussion, which is why they do not even suggest an alternative venue where these issues could be debated.

By the way, if we dig deeper into the subject, the word set intended for Forth implementers should include not only SOURCE-ID, but also QUIT. A Forth user writing application programs does not necessarily need to know how exactly the engine of a particular implementation works internally inside the word QUIT. Of course, they may know how everything is arranged purely for general education, but in application programming the word QUIT will not be needed.

As someone who has developed Forth implementations, I will say more: some words such as >IN, SOURCE, and SOURCE-ID have turned out to be unnecessary interfaces. Forth implementers do not need them as a standardized interface, because the standard itself has pushed them toward implementing the input stream as a separate stack, with its fields accessed directly. And Forth users do not use those words either, because they are not part of the application programming level. In other words, those words have become an unnecessary layer, needed by nobody. Yet the standard imposes them. Oh, what an architectural mistake! If we follow the proper path, the words >IN, SOURCE, and SOURCE-ID should be the next candidates for removal from the standard, following TIB, #TIB, QUERY, EXPECT, SPAN, and CONVERT. In turn, the standard should not dictate to implementers through which variables words such as REFILL must be implemented. This is the proper level of abstraction.

To conclude, let me summarize.

I am not proposing to deprive programmers of access to Forth’s internal mechanisms. I am proposing that these tools should not be hidden, but properly organized. All tools should remain available, but each should have its own place. The quality of the tools themselves is a separate question. For example, in my view, BASE demonstrates an inadequate level of abstraction, but that is a different issue.

The members of the standardization committee simply do not possess the expertise required to reason about the language at the architectural level. They remind me of a village tractor mechanic who knows every bolt on his machine and exactly which wrench to use to keep it running. But they are incapable of rising to a higher level of abstraction and understanding what kinds of tractors should be designed for the modern world so that agricultural work can be done efficiently.

Today, Forth looks like an untidy child whose guts are hanging out. That should not come as a surprise. Its parents resemble overgrown children who were never taught to put their toys back into the box. If, year after year, you keep throwing tools, toys, and dishes into the same cupboard, eventually the mess begins to look normal. So, guys, don't expect EuroForth to deliver anything worth your time in the coming years. It’s all down to us.

Appendix: Discussion Screenshots.

The screenshots below are included to allow readers to judge the discussion firsthand.

Discussion for BASE — Screenshot 1