DEV Community

Cover image for Copying without attention: Fourier readout of a fixed-size wave memory
Dean Chugall
Dean Chugall

Posted on Originally published at cawnai.com

Copying without attention: Fourier readout of a fixed-size wave memory

This article was first published on cawnaisclosure: an AI assistant was used for the implementation, the analys.com. More experiments are in the CAWN RSI Lab.

Abstract. A sequence model whose only memory is a fixed set of complex waves could previously locate a few important bytes in a long input and answer with one short word, but it could not copy a string from its input into its output. This work adds a read operation, called Echo, that treats the wave state as a spectrum and decomposes it over distances: for every distance dd it asks what was written dd bytes ago, selects a place with a softmax, and reads the state turned back to that place. The memory stays the same size and no attention layer is used. On a task that requires copying a short string, a tag of 6 to 12 random letters at a varying position, the loss after the first training falls from 0.3295 without Echo to 0.0301 with Echo at the same number of waves, and to 0.0049 with a wave layout whose full turn is a power of two, so that the decomposition is computed with a fast Fourier transform. Models with Echo also complete twelve generations of a two-model cooperation task through the C++ inference engine, reaching the goal in 88% (first layout) and 86% (FFT layout) of the tests at 20 steps. Echo does not solve a warehouse control task, on which the model stays at 13%; the cause of that failure is not established. A transformer with two attention layers solves the copy task with fewer parameters, and a memory of fixed size can only copy strings up to its capacity; the contribution here is that a constant-size memory can search and copy within that capacity, which is what a device with little RAM needs.

Overview in four steps: a tag stands somewhere in a long input; the input is written into a fixed set of waves of about 12 KB; the waves are read back as a spectrum with an FFT, which gives a peak at the place where the tag was written; the tag is copied into the answer, with a copy loss of 0.3295 without Echo and 0.0049 with it

1. Introduction

CAWN (Continuous Acoustic Wave Networks) layers keep no tokens. Each layer holds HH complex numbers, the waves, and every input byte is written into them. The working memory therefore does not grow with the length of the input, which matters on devices where RAM is the limit.

The post "No attention at all: a wave memory that keeps what surprised it" showed that a model built only from such layers, with no attention, can read an 11x11 map and tell a second model which way to walk, once each byte is written as strongly as a small predictor was surprised by it. That result had a narrow shape: a long input, two bytes that matter, and an answer with a single decision in it.

Most useful outputs are different. A function call contains a name or a number that stands in the input, and the model has to reproduce it byte by byte. Attention does this by keeping every token and looking it up. The question of this work is whether a memory of fixed size can do the same, and what has to be added to the layer for it.

The contributions are:

  • a read operation (Echo) that searches the wave state by content and by distance, with no additional state;
  • a controlled comparison showing that the gain comes from the read operation and not from a larger number of waves;
  • a wave layout for which the read is exactly a fast Fourier transform of length 4H4H , and an inference engine that computes it that way;
  • a statement of where the method does not help.

2. Background: the wave memory

Let xtx_t be the hidden vector of byte tt after a causal convolution. A layer computes from it a write phase φh(t)\varphi_h(t) , a query phase ψh(t)\psi_h(t) , a decay γh(t)∈(0,1)\gamma_h(t) \in (0,1) and a write gate βh(t)\beta_h(t) for each wave hh . The state is updated as

sh(t)  =  γh(t) eiθh sh(t−1)  +  βh(t) eiφh(t) s_h(t) \;=\; \gamma_h(t)\, e^{i\theta_h}\, s_h(t-1) \;+\; \beta_h(t)\, e^{i\varphi_h(t)}

where θh\theta_h is the fixed angle by which wave hh turns per byte. Unrolled, the state is a sum over everything written so far:

sh(t)  =  ∑u≤tβh(u) Γh(u,t) eiφh(u) eiθh(t−u),Γh(u,t)=∏v=u+1tγh(v) s_h(t) \;=\; \sum_{u \le t} \beta_h(u)\, \Gamma_h(u,t)\, e^{i\varphi_h(u)}\, e^{i\theta_h (t-u)}, \qquad \Gamma_h(u,t) = \prod_{v=u+1}^{t} \gamma_h(v)

The write gate follows the surprise rtr_t of a small next-byte predictor (23,010 parameters) that is trained alongside the model:

βh(t)  =  σ(gh(t)−1+wh (rt−12)) \beta_h(t) \;=\; \sigma\big(g_h(t) - 1 + w_h\,(r_t - \tfrac12)\big)

The standard readout turns each wave back by the query phase, oh=sh e−iψho_h = s_h\, e^{-i\psi_h} , and passes the real part, the imaginary part and the strength ∣sh∣2|s_h|^2 through a normalisation, a small convolution and two linear layers.

This readout looks at every wave separately. The term eiθh(t−u)e^{i\theta_h (t-u)} in the unrolled state means that a byte written t−ut-u bytes ago appears in wave hh with a rotation that depends on the distance. A fixed distance can be undone by a learned constant phase. A distance that changes from one input to the next cannot.

3. Method

3.1 The state is a spectrum

For a fixed write strength and no decay, each written byte contributes eiφh(u) eiθh(t−u)e^{i\varphi_h(u)}\, e^{i\theta_h (t-u)} to wave hh : a content phase, and a rotation that grows with the distance. Across the waves the rotation is a complex exponential in the frequency θh\theta_h , so the state is a spectrum of what was written, sampled at the frequencies θh\theta_h . Asking what was written dd bytes ago is the inverse transform: remove the content phase with a query ψh\psi_h , turn every wave back by dd steps and add them up.

R(d)  =  ∑h=0H−1sh e−iψh e−iθhd R(d) \;=\; \sum_{h=0}^{H-1} s_h\, e^{-i\psi_h}\, e^{-i\theta_h d}

For a byte at distance dd whose write phases agree with the query, all terms have the same phase and add up. For every other byte, and at every other distance, the terms have different phases and largely cancel.

Left: the amplitudes of 128 waves after two bytes were written, which show no structure. Right: the match with a query as a function of distance, with a single peak at 41 bytes, no peak at 187 where a different byte was written, and a mirror peak of opposite sign at 41 + 256

3.2 The Echo read

Echo first removes the byte that was just written, so that a search cannot find the current position:

s~h(t)  =  sh(t)−βh(t) eiφh(t) \tilde{s}_h(t) \;=\; s_h(t) - \beta_h(t)\, e^{i\varphi_h(t)}

It then runs two searches, j=1,2j = 1, 2 . Each search has a query phase ψj,h\psi_{j,h} and a learned weight per wave aj,ha_{j,h} , and computes a score for every distance d=1,…,Nd = 1, \dots, N :

Sj(d)  =  eτjH  Re⁡∑h=0H−1aj,h s~h e−iψj,h e−iθhd S_j(d) \;=\; \frac{e^{\tau_j} }{\sqrt{H} }\; \operatorname{Re} \sum_{h=0}^{H-1} a_{j,h}\, \tilde{s}h\, e^{-i\psi{j,h} }\, e^{-i\theta_h d}

Inserting the unrolled state shows what the score measures. The byte written at distance dd contributes ∑haj,h βh Γhcos⁡(φh(t−d)−ψj,h)\sum_h a_{j,h}\, \beta_h\, \Gamma_h \cos(\varphi_h(t-d) - \psi_{j,h}) , which is large when the query phases agree with the write phases of that byte across the waves. All other bytes contribute terms with phases that depend on the wave and average out. τj\tau_j is a learned sharpness.

A softmax over the distances selects a place, and the state is turned back to it:

qj(d)=eSj(d)∑d′eSj(d′),ρj,h  =  s~h∑d=1Nqj(d) e−iθh(d−δj) q_j(d) = \frac{e^{S_j(d)} }{\sum_{d'} e^{S_j(d')} }, \qquad \rho_{j,h} \;=\; \tilde{s}h \sum{d=1}^{N} q_j(d)\, e^{-i\theta_h (d - \delta_j)}

The two searches differ in the query and in the offset δj\delta_j :

  • Search 1 uses the write phase of the current byte as its query, ψ1,h=φh(t)\psi_{1,h} = \varphi_h(t) , and reads one place later, δ1=1\delta_1 = 1 . It answers the question "where was this byte seen before, and what followed it". The query matches from the first training step, because it is the same function that produced the write phase.
  • Search 2 uses a learned query, ψ2,h=πtanh⁡(Wext)h\psi_{2,h} = \pi \tanh(W_e x_t)_h , and reads the place it finds, δ2=0\delta_2 = 0 .

The four vectors Re⁡ρ1,Im⁡ρ1,Re⁡ρ2,Im⁡ρ2\operatorname{Re}\rho_1, \operatorname{Im}\rho_1, \operatorname{Re}\rho_2, \operatorname{Im}\rho_2 are normalised and passed through two linear layers, and the result is added to the output of the standard readout:

et  =  Wout SiLU⁡!(Win RMSNorm⁡([Re⁡ρ1,Im⁡ρ1,Re⁡ρ2,Im⁡ρ2])) e_t \;=\; W_{\text{out} }\, \operatorname{SiLU}!\big(W_{\text{in} }\, \operatorname{RMSNorm}([\operatorname{Re}\rho_1, \operatorname{Im}\rho_1, \operatorname{Re}\rho_2, \operatorname{Im}\rho_2])\big)

Block diagram of one CAWN layer: the byte passes a causal convolution and projections; a small predictor supplies the surprise that opens the write gate; the wave state is read by the standard readout and, in parallel, by the Echo read with its two searches; both are added and passed to the feed-forward block

Echo keeps no state of its own. It reads the same HH complex numbers that the layer already holds. Every one of the six layers has its own Echo read.

Two rows of byte cells. In the first, at the start of the answer, a learned query finds the first letter of the tag in the input and that place is read. In the second, in the middle of the answer, the byte just written is found in the input and the place right after it is read

3.3 How many waves are needed

When the state holds MM strongly written bytes, the peak of a matching byte is of the order βH\beta H , while the other M−1M-1 bytes add terms of random phase with a standard deviation of the order βMH/2\beta \sqrt{MH/2} . The ratio is

peaknoise  ≈  2HM \frac{\text{peak} }{\text{noise} } \;\approx\; \sqrt{\frac{2H}{M} }

For a copy of about ten letters, with the letters of the answer written as well, M≈20M \approx 20 . This gives about 1.8 for 32 waves, 2.5 for 64 and 3.6 for 128. The estimate ignores decay and unequal write strengths; it is used only to choose the sizes tested below. It is the practical side of a known result: a model with a state of fixed size cannot copy strings beyond what that state can hold (Jelassi et al., 2024). Echo does not remove that limit. It lets the model use the capacity it has.

3.4 A layout for which the read is an FFT

The first layout used the angles θh=π(h+1)/(H+1)\theta_h = \pi (h+1)/(H+1) . All waves return to their starting position after 2(H+1)2(H+1) bytes, so the scores repeat with that period: 258 bytes for 128 waves. With this layout the score and the read were computed from tables of cos⁡(θhd)\cos(\theta_h d) and sin⁡(θhd)\sin(\theta_h d) , at a cost of O(HN)O(HN) per search.

Shifting the angles by half a step,

θh  =  π (h+12)H  =  2π (2h+1)N,N=4H \theta_h \;=\; \frac{\pi\,(h + \tfrac12)}{H} \;=\; \frac{2\pi\,(2h+1)}{N}, \qquad N = 4H

makes the full turn exactly N=4HN = 4H bytes, a power of two when HH is (128, 256 and 512 bytes for 32, 64 and 128 waves). Wave hh is then bin 2h+12h+1 of a discrete Fourier transform of length NN . With c2h+1=aj,h s~h e−iψj,hc_{2h+1} = a_{j,h}\, \tilde{s}h\, e^{-i\psi{j,h} } and all other ck=0c_k = 0 ,

Sj(d)  =  eτjH Re⁡ Cd,Cd=∑k=0N−1ck e−2πi kd/N S_j(d) \;=\; \frac{e^{\tau_j} }{\sqrt{H} }\, \operatorname{Re}\, C_d, \qquad C_d = \sum_{k=0}^{N-1} c_k\, e^{-2\pi i\, k d / N}

and the sum needed for the read is the conjugate of an odd bin of the transform of the softmax weights:

∑dqj(d) eiθhd  =  Q2h+1‾,Qk=∑d=0N−1qj(d) e−2πi kd/N \sum_{d} q_j(d)\, e^{i\theta_h d} \;=\; \overline{Q_{2h+1} }, \qquad Q_k = \sum_{d=0}^{N-1} q_j(d)\, e^{-2\pi i\, k d / N}

Each search is two transforms of length NN , at O(Nlog⁡N)O(N \log N) , and the tables are no longer needed. Because only odd bins are used, the score has the property S(d+N/2)=−S(d)S(d + N/2) = -S(d) : a byte produces a positive peak at its distance and a mirror peak of opposite sign half a turn later, visible in the figure of section 3.1. The softmax selects positive peaks, so the mirror does not act as a second match.

In training, both layouts use the tables. In the inference engine the first layout uses tables and the second uses the FFT.

4. Experimental setup

Models. All CAWN models have six layers of width 128, four heads, no attention layer, a byte vocabulary, and are trained from random weights. The number of waves per layer is H=4KH = 4K with KK = 8, 16 or 32. The references are transformers with two and six attention layers of the same width.

Map task. As in "No attention at all: a wave memory that keeps what surprised it": the input holds an 11x11 room with a walker and a goal, and the model answers with one of four made-up words. The loss of a model that cannot read the map is ln⁡4/43=0.0322\ln 4 / 43 = 0.0322 .

Copy task. The input is the same, preceded by id, a tag of 6 to 12 random lowercase letters and 0 to 60 filler characters; the answer must contain the tag. Over 2,000 generated lessons the tag stands 191 to 257 bytes before its copy. A model that produces the fixed part of the answer and guesses the letters has a loss of about nˉln⁡26/(41+nˉ)=0.59\bar{n} \ln 26 / (41 + \bar{n}) = 0.59 for the mean tag length nˉ=9\bar{n} = 9 ; a loss under 0.05 means that the tag is copied. An earlier version of the task with a tag of fixed length at a fixed distance was solved by the model without Echo (loss 0.0009 at the third epoch) and was therefore discarded: a fixed distance can be undone by a constant phase.

First-training check. For both tasks a model is trained once on about 9,300 lessons for 8 epochs at a learning rate of 5e-4, and the final training loss is reported.

Full run. Twelve generations of the task from "One sees, one walks: two tiny models learn to talk with no teacher": a guide that sees the room and a walker that sees 5x5 cells learn from episodes that reached the goal; the goal moves 2 steps further away whenever 70% of 100 fixed test tasks are reached. In the full run the models are exported to int8 and played by the C++ engine.

Hardware. One RTX 2060 SUPER (8 GB). Several jobs often shared the GPU, so training times are indicative only.

5. Results

5.1 Copying

Dot plot on a logarithmic scale of the loss on the copy task: 32 waves without Echo 0.4954, 128 waves without Echo 0.3295, the first Echo version with 32 waves 0.4731, Echo with 64 waves 0.0718, Echo with 128 waves and tables 0.0301, Echo with 128 waves and the FFT layout 0.0049, and a transformer with two attention layers 0.0045 at the third of eight epochs

Model Waves per layer Echo Copy loss
CAWN, writing by surprise 32 no 0.4954
CAWN, writing by surprise 128 no 0.3295
CAWN, first Echo version 32 yes 0.4731
CAWN, Echo, first layout 64 yes 0.0718
CAWN, Echo, first layout 128 yes 0.0301
CAWN, Echo, FFT layout 128 yes 0.0049
Transformer, 2 attention layers – – 0.0045 (epoch 3 of 8)

Loss after the first training, seed 0. For the transformer only the value at the third epoch was recorded.

Without Echo the model does not copy, and four times as many waves change little (0.4954 to 0.3295). With Echo and the same 128 waves the loss is 0.0301, eleven times lower. Echo with 64 waves (0.0718) is better than 128 waves without it. A second seed of the 128-wave Echo model was at 0.0319 after half of its training when it was stopped; the first seed was at 0.0424 at three quarters and ended at 0.0301.

The loss falls with the number of waves, 0.0718 at 64 and 0.0301 at 128, in the direction the estimate of section 3.3 predicts.

Ablation: the first Echo version. A version with 32 waves and a single search, whose query was the query phase of the standard readout and whose state included the byte just written, did not improve on the baseline (0.4731 against 0.4954). Two causes are consistent with the analysis: 32 waves give a peak-to-noise ratio below 2, and a query that has to be learned from nothing gives the softmax no signal at the start of training.

5.2 The FFT layout

With the FFT layout the copy loss is 0.0049, six times lower than with the first layout. The cause has not been established. Three things changed between the two runs: the layout of the angles (and with it the full turn, 512 instead of 258 bytes), the tables were computed in double precision, and the per-wave weights aj,ha_{j,h} were no longer subject to weight decay.

Bars on a common axis of bytes: the first layout repeats after 258 bytes, the FFT layout after 512; the tag stands 191 to 257 bytes behind its copy; a whole lesson is up to 378 bytes long

One explanation can be excluded. The tag is at most 257 bytes behind its copy, inside one full turn of both layouts, so the first layout did not confuse the position of the tag with a nearer one. A whole lesson is up to 378 bytes long and does exceed 258; whether the bytes beyond one turn disturb the search with the first layout was not measured. The result rests on one seed.

5.3 Agreement of training code and inference engine

The training code computes Echo from tables in PyTorch; the engine is separate C++ code. Both were run on the same model with random weights and a 234-byte input.

Configuration Largest difference of the logits Same next byte
no Echo, 128 waves 0.0002 100.0%
Echo, first layout (tables), 128 waves 0.0023 99.6%
Echo, FFT, 128 waves 0.0015 100.0%
Echo, FFT, 64 waves 0.0018 100.0%
Echo, FFT, 32 waves 0.0005 100.0%
Echo, FFT, 128 waves, engine on 4 threads 0.0015 100.0%

fp32 weights; the largest logit is about 3. For one byte in the first layer, the place selected by each search and its softmax weight agree between the two implementations to five or six digits.

The larger difference with Echo comes from the sharp selection of a place, which magnifies rounding through six layers: with the softmax flattened the difference returns to 0.0002. Engine time per byte on one x86 thread with fp32 weights was 0.44 ms without Echo, 0.52 ms with tables and 0.44 ms with the FFT.

5.4 Reading the map

Bar chart of the guide loss on the map task: 32 waves without Echo 0.0044, 128 waves without Echo 0.0031, 64 waves with Echo 0.0031, 128 waves with Echo and tables 0.0017, 128 waves with Echo and the FFT layout 0.0029, transformer with two attention layers 0.0075; the blind level is 0.0322

Every Echo variant reads the map, far below the blind level of 0.0322. More waves alone lower the loss from 0.0044 to 0.0031; Echo with the first layout lowers it further to 0.0017, and with the FFT layout it is 0.0029. With one seed each, the differences among the last four values are not large enough to rank them. The first Echo version also read the map (0.0056).

5.5 Twelve generations through the engine

Two line charts over twelve generations. Share of test tasks reached: the Echo model with the FFT layout starts at 90%, stays between 80% and 92% and ends at 86%; the Echo model with the first layout falls to 74% at generation 3 and ends at 88%; the two runs without Echo end at 89% and 82%; the transformer with six attention layers ends at 84%. Bits per word rise from between 0.3 and 0.4 to between 1.0 and 1.3

Model Attention layers Reached at 20 steps Lowest in the run Bits per word
CAWN with Echo, FFT layout, 128 waves 0 86% 80% 1.17
CAWN with Echo, first layout, 128 waves 0 88% 74% 1.34
CAWN, no Echo, 32 waves, run 1 0 89% 77% 1.17
CAWN, no Echo, 32 waves, run 2 0 82% 74% 1.26
Transformer 6 84% 83% 0.99

Last generation; 100 fixed test tasks per generation. "Bits" is the information the guide's word carries about the walker's next move; 2.00 is a perfect code for four moves.

Both Echo models moved the goal at the same generations as the others, from 4 to 20 steps. With the first layout the run ended at 88% and 1.34 bits per word; between generations 2 and 4 it was lower than the model without Echo (76%, 74% and 78% against 84%, 82% and 80%) and recovered afterwards. With the FFT layout the run ended at 86% and 1.17 bits and did not show that dip (83%, 83% and 80%). With the guide silent the walker reached between 0% and 10% of the goals in every generation of both runs.

With 100 test tasks, differences of a few points are noise, so the five runs cannot be ranked. On this task Echo is neither required nor harmful. What the runs show is that models with Echo train over many generations and play correctly through the int8 engine, with the read computed from tables in one run and by the FFT in the other.

6. Cost

Model Waves per layer Parameters int8 file State
CAWN, writing by surprise 32 1,222,914 1.37 MB about 8 KB
CAWN, writing by surprise 128 2,039,682 2.22 MB about 12 KB
CAWN with Echo 64 1,843,470 3.05 MB about 9 KB
CAWN with Echo 128 2,636,430 4.60 MB about 12 KB
Transformer, 2 attention layers – 395,264 – grows with the input

Parameters and file sizes are from built and exported models. The state is computed from the sizes of the arrays (waves and convolution history of six layers), not measured on a device.

The Echo matrices are currently stored as fp32 inside the int8 file. Stored as int8 they would bring the files to about 2.01 MB (64 waves) and 2.83 MB (128 waves); these two figures are computed, not exported. With the first layout the engine also holds four tables of about 0.5 MB in total for 128 waves; with the FFT layout these are replaced by a table of N/2N/2 complex roots of unity.

A first training of the 128-wave Echo model took about twice as long as that of the 32-wave model without Echo.

The number of waves is the size parameter of the method. On the measurements above, 32 waves are enough for the map task, 64 waves copy with a visible error, and 128 waves copy a tag of about ten letters.

7. Where a constant memory matters

This section states what the method is for. Only the tasks of section 5 were measured; the applications below are the settings that those measurements point to.

Working memory against input length on logarithmic axes: two and six attention layers grow in proportion to the input and pass the 400 KB of an ESP32-C3 at about 390 and 130 bytes; the wave state stays at about 12 KB for any length

In the inference engine used here, an attention layer of this width keeps 512 bytes of keys and values for every input byte (two key-value heads of 32 dimensions in fp32), so its working memory is

Mattention(n)  =  512⋅L⋅n bytes,Mwaves(n)  ≈  12 KB M_{\text{attention} }(n) \;=\; 512 \cdot L \cdot n \ \text{bytes}, \qquad M_{\text{waves} }(n) \;\approx\; 12\ \text{KB}

for LL attention layers and an input of nn bytes. Two attention layers pass 400 KB, the whole RAM of an ESP32-C3, at about 390 bytes of input. The copy lessons of this work are up to 378 bytes long. These figures are computed, not measured on a device, and a transformer can be given a shorter window at the price of forgetting what lies outside it.

Property Transformer CAWN with Echo Basis
Working memory grows with the input about 12 KB computed
Longest input the window that fits in RAM no limit from memory by design
Work per input byte grows with the context constant by design
State to save for pause and resume all keys and values about 12 KB computed
Parameters, copy task 0.4 million 1.8 to 2.6 million measured
Copy loss 0.0045 (epoch 3 of 8) 0.0049 to 0.0718 measured
Longest string copied limited by the window limited by the waves sec. 3.3

The advantage of the wave memory is in the first four rows; the transformer is ahead in the next two.

The settings in which the first four rows decide:

  • Function calling on a microcontroller. The input is a list of tools and a request; the output is a call that contains a name or a number from the request. Before Echo, a model without attention could choose among fixed answers but could not put a value from the input into its output. This is the application the copy task stands for; it has not been tested yet.
  • Reading a long stream and reporting from it. A serial log, bus traffic or meter readings arrive without end. Writing by surprise keeps the lines that could not be predicted, and Echo can retrieve an identifier or a value from them when a report has to be written.
  • Devices that run for days. The state does not grow, so no window has to be cut and nothing has to be re-read after a pause; the whole state can be stored in about 12 KB and restored.
  • Several models on one chip. With about 12 KB of working memory each, several small models fit beside each other where one transformer with a useful window would not.

The method is not a replacement for a transformer where RAM is plentiful: there a small transformer copies more accurately with fewer parameters.

8. Related work

Complex-valued associative memories. Holographic reduced representations store bound pairs in a single vector. Associative LSTM (Danihelka et al., 2016) keeps key-value pairs in a complex vector of fixed size, binds by element-wise complex multiplication and retrieves by multiplying with the conjugate key, reducing noise with redundant copies. The write and the content match of this work are of the same family. Associative LSTM has no decomposition over distance and no selection of a place.

Histories stored as coefficients. HiPPO (Gu et al., 2020) and the Legendre Memory Unit keep a state of fixed size that holds the coefficients of the input history in an orthogonal polynomial basis, from which the history can be approximately reconstructed at any point. Fourier Recurrent Units (Zhang et al., 2018) summarise the hidden state in a Fourier basis. The view of the state as a spectrum is shared with these works; they do not search the reconstructed history by content.

Fourier transforms in place of attention. Autoformer (Wu et al., 2021) computes an auto-correlation with an FFT and aggregates delayed sub-series with a softmax over time delays, which is close to the selection step used here, but over a series that is kept in full. FFTNet (2025) mixes tokens with an FFT along the sequence and also keeps all tokens.

Selection and copying in models with a fixed state. Mamba (Gu and Dao, 2023) uses a selective copying task to show that input-dependent gating is needed for it, which corresponds to the surprise-driven write gate used here. Jelassi et al. (2024) prove that a two-layer transformer can copy strings of exponential length while models with a latent state of fixed size are limited by that size; the dependence on the number of waves in section 5.1 is an instance of this limit.

A search of these lines of work did not find the combination used here: a recurrent state of complex phasors, searched by content through an inverse Fourier transform over distances with a softmax over the distances, used for copying. The search was limited to a small number of queries, and no claim of priority is made.

9. Limitations

  • A small transformer does the same with fewer parameters. Two attention layers and 395,264 parameters reach 0.0045 on the copy task by the third epoch. The Echo model has 2.6 million parameters. What it offers is a working memory that does not grow with the input.
  • The warehouse task is not solved. In the task from "A 5M-parameter model that teaches itself to run a warehouse" the Echo model was at 13% after 63 generations (best 21%), where a small hybrid model (CAWN layers with two attention layers among them) reached 76%. The cause has not been established. Fields with few possible values receive a low surprise and are written weakly, and a variant in which half of the waves write every byte reached 39% at generation 16 without Echo, which points to the write rule. Against this, the walker of the talking task relies on a field with only four possible values and reads it without difficulty, so a low surprise alone does not explain the failure. A read operation cannot recover what was not written, but whether that is what happens here was not tested.
  • The tag is random. Random letters are maximally surprising and therefore strongly written. Copying a name or a number inside a function call was not tested.
  • Mostly one seed. The control, the three sizes, the FFT layout and each full run rest on one seed; a second seed exists only for the 128-wave model with the first layout and was stopped at half of its training.
  • The gain of the FFT layout is unexplained (section 5.2).
  • Training loss, not test accuracy, on the copy task. The copy task was evaluated by the loss of the first training. Only the talking task was played end to end.
  • No measurement on a device. Memory figures are computed. The engine was timed on an x86 processor; a microcontroller without a floating-point unit would need a fixed-point implementation.
  • Capacity. A state of fixed size bounds the length of what can be copied (section 3.3 and Jelassi et al., 2024). Strings longer than about ten letters were not tested.
  • The literature search was short (section 8); related work may have been missed.
  • Language modelling was not measured.

10. Conclusion

A fixed set of complex waves stores what was written as a spectrum. Reading it by an inverse Fourier transform over distances, with a softmax that selects a place, gives a model without attention the ability to find a string in its input and copy it, at no cost in state. With an angle layout whose full turn is a power of two, the read is two fast Fourier transforms per search. The method can only read what was written; one control task on which the model fails is reported, with its cause open.

The next steps are a function-calling task trained from random weights without attention, further seeds for the FFT layout, the cause of the failure on the warehouse task, int8 storage of the Echo matrices, and a fixed-point engine for a microcontroller. More experiments are in the RSI Lab.


Project site: cawnai.com · Lab: cawnai.com/lab · Blog: cawnai.com/blog

Top comments (0)