Forth’s Architectural Mess: The Case of the Numeric Output Words

This article is a logical continuation of the article “Changes to the Standard, or, How Europe's Bush-Leaguers Murder Forth.”

The idea in this new article is the same: Forth has now become an architectural mess. The problem lies in the language standard—or, more precisely, in the standards committee, whose members still have not understood which path the programming language should have taken. As a result, Forth has lost much of its standing in the global programming community. In contrast to the committee’s actions, I argue that a sound language architecture not only teaches Forth programmers to design programs properly, but also attracts new programmers to Forth. We will examine the issue using the numeric output words as an example.

Historical Forth mixes several different things together. The numeric output words simultaneously encode:

1. the size of the number — single or double;
2. signedness — signed or unsigned;
3. the number base — BASE;
4. field width — for .R;
5. the actual algorithm for converting a number into a sequence of digits.

Architecturally, however, this is the wrong approach: most of these distinctions should not give rise to different primitives.

The shortcomings of the current standard are rooted in its history. Forth was created as an interactive system. A person works at the console and types:

HEX ABCD FFFF DECIMAL 1000

In other words, changing the number base is a property of the working session, not a separate function. This is excellent for a REPL. For libraries, it is not so good.

Let us examine the problem using BASE as an example. Today, it is implicit global state. By itself, this is convenient. The problems begin when BASE becomes part of the execution context. Words cease to be pure: the behavior of a word depends not only on the stack, but also on external global state.

In functional programming, one would say that BASE is a hidden argument to the function. The problem is implicit input data. Modern design practice favors simple composition of words without hidden dependencies on global state:

f(x, base)

instead of

global base
f(x)

because the first form is local and explicit.

Historically, Forth conflated two completely different tasks:

• how to interpret text (123, ABCD, etc.);
• how to present a number to the user.

BASE handles both at once. This was very convenient in the era of teletypes and interactive computing, but architecturally it couples two independent mechanisms.

To completely redesign the current standard, these two things need to be separated:

1. The input base for the REPL — global state of the interpreter.
2. The formatting base — an explicit parameter.

The proper architecture for a new standard should look like this:

INPUT-BASE — affects only the text interpreter;
FORMAT — explicit conversion of a number into a string;
TYPE — output of the string;

And the word “dot” is simply shorthand for the sequence of words:

DECIMAL FORMAT TYPE

At the same time, one could write

123 HEX FORMAT TYPE

without changing any global state. This preserves Forth’s conciseness when using the word “dot” while eliminating one of the language’s most unpleasant hidden dependencies.

If we look deeper into the problem of having several numeric output words in the standard, we encounter an interesting philosophical problem: the number of words in a dictionary is often a symptom not of a language’s richness, but of poorly chosen abstractions.

Applied to Forth: the less successful the language model, the more words it needs as exceptions.

Today, the words U., D., U.R, and D.R look like exceptions to a general rule. Historically, this is understandable, but conceptually they are simply different entry points into the same task. Why not have a single word? For example:

123 PRINT

And that's it. If formatting and layout are needed, then:

123 HEX 8 WIDTH PRINT

In this case, the stack describes intent, rather than selecting a specialized word.

Applied to Forth, it would be more appropriate to write it like this:

123 HEX FORMAT TYPE

Architecturally, this is cleaner. At the same time, no one is taking aim at the historical word “dot,” because it follows from the general rule:

: . DECIMAL FORMAT TYPE ;

Forth contains many historical “paired” words. Some of them arise because the language has no unified model of an object or data type. If there were a more rigorous concept of “value → representation,” half of these exceptions would disappear.

Let us establish the proper understanding once and for all: good Forth is not about having the minimum number of words. It is about having the minimum number of rules from which the words are derived.

In this sense, FORMAT as a separate layer looks much more Forth-like than a collection of U., D., and so on. We are not adding a new word for every special case; we are building a small set of basic operations whose combinations cover all cases.

Historical Forth contains U., D., U.R, D.R, and .. Every new case requires a new word. The problem is not the number of words. The problem is that each word contains several decisions at once.

Today, the words in Forth have lost their purity, but suppose the model were:

NUMBER FORMAT WIDTH ALIGN TYPE

Then the user does not learn the separate concept of U.R. The user learns several general concepts that can be combined.

Good Forth should look like a small dictionary with a large space of possibilities. The dictionary may be large as well, but not because of new exceptional cases.

The bad approach is:

U. D. U.R D.R .

because each word answers the question, “Which particular way of printing?”

The good approach is:

FORMAT WIDTH ALIGN TYPE

because the words answer the question, “Which properties of the number’s representation?”

Of course, an adept of the old architecture will say that Forth is already minimalist enough as it is. But let us look at the fact that the language has not become any better for it: you have simply hidden the absence of a model behind names. We are not talking about composition at the implementation level, but about composition at the level of the language’s semantic model. The mere fact that a word is built from primitives says nothing about the quality of the abstraction. The key question is whether the word is a new building block, or merely a patch over a crack in the architecture.

We now move naturally to the question of which words should exist in the language as primitives, and which should be expressed in terms of them.

Let us follow these principles:

1. The dictionary remains extensible, but the standard should describe the core, not every possible library.
2. Minimize hidden state. In this case, that state is BASE.
3. Words should express concepts, not cases. If a new requirement calls for adding a new word, perhaps the abstraction is missing something.
4. A new level of abstraction should not explain what already exists; it should eliminate existing exceptions.

Let us start with a simple question: what does the word “dot” do? It appears to be a single action. But in reality, several different things happen inside it:

1. The number is taken from the stack.
2. Its interpretation is determined: signed? unsigned? single? double?
3. The base is selected: DECIMAL? HEX?
4. The number is converted into a sequence of characters.
5. The characters are output.

In other words, this is not simply number output. It is a small combine harvester.

At what point does the problem begin? A requirement appears:

123 .

Fine. But then another requirement appears:

123 U.

Why? Because it turns out that the word “dot” did not merely output the number; it made a decision about the number’s type.
Then comes 123 D. as another exception.
Then 123 U.R — another one.
Then 123 D.R — yet another exception.
Now imagine that tomorrow we need:

• output in HEX;
• output with leading zeros;
• output with thousands separators;
• fixed-point output.

Where do we add them? In the end, this becomes a path of endless exceptions.

Now consider the proper model. Separate the concepts. A number is a number. The representation of a number is a format. Output is output.

In the new model:

: U.R ( u width -- )
    >R
    10 FORMAT
    R> WIDTH RIGHT
    TYPE
;

In other words, U.R is not a fundamental mechanism. It is simply a convenient combination.

And here a question arises: “Should the word U.R be part of the language, or is it simply the first convenient program written by a user?”

Taking this further, the question becomes: “What should be in the core, and what should belong to the layer of convenient extensions?”

A small combine harvester is a useful thing, but no sane person would think of building a combine harvester into a hammer.

And now the answer to the questions we have asked is this: “The core should contain not a combine harvester, but several simple mechanisms.”
And where should the “combine harvesters” that Forthers have long been accustomed to live? They should live in a special set of words containing canonical recipes, called PATTERNS. Strictly speaking, it would be more appropriate to call this set of words IDIOMS. An idiom is not only a linguistic term; it is also a programming term. An idiom is an established way of solving a problem using the means provided by a language. But since most programmers are often simply technicians, the term PATTERNS may be more suitable for them.

Here are the ready-made recipes from PATTERNS:

. U. D. U.R D.R .

These are not language primitives. They are canonical examples of dictionary extensions. And now for the most important point: we are not banning short words; we are proposing that combine harvesters should not be part of the core. In other words, we are not removing convenient words. We are removing the need to know dozens of special cases.

In the wrong model, every new need gives rise to a new word. In the right model, the combination of VALUE + FORMAT + OPTIONS + OUTPUT covers everything.

In fact, we are not arguing about the size of the dictionary, nor are we arguing about syntax or compatibility. We are asking just one question about every word:

“Does this word add a new semantic concept, or does it merely fix one of the combinations of concepts that already exist?”

If it is the latter, then it is a candidate for PATTERNS, not for CORE.

But this raises the next architectural question. If historical words are merely combinations of more fundamental operations, then we must have a clear understanding of exactly what those operations combine.

We need to identify the object being transformed: the numeric value has already been determined, but how should it be represented in textual form?

This is fundamentally different from interpreting the number itself. For example, a sequence of bits can be interpreted as a signed or unsigned value. This is a question of the semantics of the value. But once the value has been unambiguously determined, it can be represented in different bases, with different alphabets, and in different forms. This is already a question of representation policy.

This is where the architectural concept of Notation appears—not as yet another special word for outputting a number, but as a way of describing the rules by which a numeric value is transformed into a textual representation.

It is more natural to implement this concept not necessarily as a separate object on the user stack, but as a representation policy used by the number-conversion mechanism called REPRESENT. The concept of Notation is used to denote this policy.

Notation describes exactly how a value should be represented: in which base, in what form, using which alphabet, and with what precision. A concrete machine representation of such a policy can be compactly encoded in a descriptor. Thus, Notation is an architectural concept, while a descriptor is its concrete representation, convenient for passing and processing.

This makes it possible to separate the representation policy itself from the numeric value and from the mechanism that outputs it. At the same time, Notation does not have to be a separate language object: what matters is not where the policy is physically stored, but that it becomes an explicit and orthogonal level of semantics.

An approximate orthogonal model looks like this:

NUMERIC VALUE
│
│ interpretation
▼
REPRESENTATION POLICY
│
├── RADIX
├── POINT
├── ALPHABET
└── PRECISION
│
▼
REPRESENT
│
▼
TEXT REPRESENTATION

SIGNED/UNSIGNED are deliberately not properties of the representation policy here. They determine the interpretation of the numeric value, not the way it is represented as text.

Thus, instead of a multitude of special-purpose words, we obtain a small orthogonal model of representation policy. It describes not a specific output operation, but the independent properties of how a value is to be represented. Various representation algorithms and different methods of subsequent output can then be built on top of it.

We are not claiming that this is the final REPRESENT API. At the same time, we have not merely assembled a convenient descriptor. We have applied the following criterion:

“If changing a parameter changes the mathematical value, it is not a representation parameter. If the value has already been determined and the parameter can be changed without changing that value, it is a candidate for the representation policy.”

A complete description of this model, including the specific structure of the representation policy, the REPRESENT algorithms, and their implementation, is beyond the scope of this article. What matters here is to establish the architectural principle itself: the value, its representation, and its output are different levels of semantics.

If some newcomers to programming find parts of the above unclear, then let’s start with the basics: get acquainted with the term “orthogonality.” This is one of the fundamentals of modern programming, and it is well worth getting to grips with. The trouble with Forth today is that its standardization committee is made up of poorly qualified programmers who cannot even command terminology at the level of language architecture.

As a developer who implemented my own Forth many years ago and has now undertaken a new implementation, I can see all the shortcomings of the current standard. What was forgivable in the 1994 standard, where some words were retained for compatibility, became a crime in the 2012 standard, which incorporated all of the previous shortcomings. There are so many flaws in the current standard that I cannot confidently say I will be able to finish my project, because I have to shoehorn myself into the nonsense on which the standard is based.

I have read the criticism directed at me following the first article. For example, there was the issue of the >IN word, which is supposedly used in application programs. First of all, let us define what we mean by application programs. Modern languages and ecosystems allow application programmers to work at a high level of abstraction without directly interacting with most of the details of program execution. Second, in my first article I briefly mentioned an implementation of the input stream as a separate stack, operating through an Input Source Frame (called different things in different implementations). Here again, the standards committee lumped everything together, leaving words that belong closer to the interpreter’s internal workings for users to deal with. Is that not why there are so few user applications written in Forth worldwide—that the standard itself is fundamentally unattractive for application development? And if the committee standardizes the internals of the input stream as part of its public interface, then that is a problem with the committee, not with the programming language.

The other issue is that the wrong tool has produced the wrong way of thinking in some of its users. Yes, they like such a tool, and they say so: “I like it.” But the problem is that such users are not the driving force behind Forth. What is more, they are destroying Forth, because the programming language has become unattractive to the global community and has turned into a tool for outcasts. And that is already a separate subject, beyond the scope of programming. Paraphrasing a well-known saying about sports, one might put it this way: “They do not love Forth in themselves; they love themselves in Forth.” If Forth did not exist, they would find another wrong tool and praise it to the skies, because they like wrongness in themselves. Guys, this is beginning to smell like some kind of cult.

All right, one could still argue about the word >IN, but another example is the word BASE. Let us return once again to the subject of language architecture. Let us decide what we should be putting on the stack. The word BASE leaves an address on the stack. Now ask yourself: does the user need to see this address, for example, through the word .S? Consider the absurdity of the following combination:

BASE .S @

Can you imagine the user needing to see the address of the current number base on the stack? No, of course not. And why? Because the address on the stack is an intermediate link that nobody is interested in: immediately after using BASE, one uses either @ or !. And if an address is an intermediate state, it has no business being on the stack.

So what is the word BASE doing in the CORE word set in its current form? The question is rhetorical, because it is there like a scarecrow in a garden. This is an architectural flaw in the BASE interface.

Such exposed internals scare away any potential newcomers who could become the foundation of the Forth community. Instead, our programming language looks like a piece of junk to the entire global community. It is astonishing that the cultists boast about using a two-word combination instead of a single word that could immediately return the current number base to the stack. This is what a bad standard does: it raises a generation of programmers with distorted thinking.

Some people, of course, are already accustomed to working through the BASE @ combination, but that is because they were taught from the very beginning to take the scenic route. There is a nice ironic idiom in English for this: “to take the scenic route” — literally, to go the long way around. And that is exactly what they do: they go the long way around instead of taking the straight road.

Architecture runs throughout this article, and for me, the science of design as applied to Forth conjures up the physical image of an abandoned building somewhere on the outskirts of town. That building is run by a horde of street urchins led by a small-time prima donna. And if Forthers do not want to look like a marginalized group in the modern world, they need to stop clustering around that architectural structure, patched and repatched because it was fundamentally misdesigned from the outset.

As for the standardization committee, there is a familiar saying that has become a catchphrase: “You can take the boy out of the country, but you can’t take the country out of the boy.”