Large language models can produce language that feels coherent. They can summarize an argument, imitate a style, explain a concept, and continue a pattern with remarkable fluency.
But fluency leaves a harder question unanswered: has the model learned a rule it can apply systematically, or has it become exceptionally good at recognizing familiar statistical patterns?
That distinction is the question behind compositionality.
Human thought is compositional because we can combine known parts into new wholes. If we understand "the dog chased the cat," we can understand "the cat chased the dog." The vocabulary stays the same, but the relations change. We can also understand sentences we have never heard before because we do not need to memorize every possible combination. We reuse structures and rules.
This ability matters far beyond grammar. A system that represents rules should be able to carry them into unfamiliar cases. A system that relies mainly on resemblance may perform very well until the vocabulary, structure, or context moves away from familiar examples.
For ordinary writing assistance, that difference may not always be important. For systems expected to reason, explain decisions, write reliable code, or operate outside their training distribution, it becomes much harder to ignore.
Two competing views
In my research on the philosophy of artificial intelligence, I examined this problem through a debate between two broad positions.
The emergence view, associated with modern connectionism, argues that sufficiently complex cognitive abilities can arise from deep learning. On this account, models do not need a separate symbolic engine designed into them. With enough data, scale, and the right training, structured behavior can emerge from the network itself.
The competing view, associated with Gary Marcus and the symbolic tradition, argues that statistical learning runs into a "compositionality wall." Reliable abstraction requires structured representations of objects, relations, hierarchy, and rules. A model may become highly fluent without gaining the kind of systematicity that lets a rule survive across genuinely new cases.
I tested a narrower version of this debate on Llama 3 8B. The full study used three controlled experiments. Two examples capture the central finding.
Example one: invented words
The first test used sentences built from meaningless words.
Why use nonsense vocabulary? Because familiar words carry a great deal of hidden help. A model can draw on common phrases, world knowledge, and statistical associations between known terms. Invented words remove most of those clues. The model has to judge whether the structure itself is valid.
I presented 50 grammatically valid sentences and 50 invalid ones. For example:
- Valid: "A wug poons wugs." The invented singular subject wug matches the invented singular verb poons.
- Invalid: "Wugs poons wugs." The plural subject wugs is paired with the singular verb poons.
The words carry no ordinary meaning. The decision should turn on subject-verb agreement alone.
The model correctly identified invalid sentences with a proper explanation in 66% of cases. But it rejected valid, unfamiliar sentences as wrong in 64% of cases. Of the 36% it accepted, 32% of the full set received a fully correct judgment and explanation, while 4% were only partially successful.
That asymmetry matters. The model was much better at detecting something unusual than at recognizing that a new example followed the rule. It often treated unfamiliarity as evidence of error.
A system with an abstract grammar should work in both directions. It should reject violations, but it should also accept a new construction when that construction obeys the rule. The model behaved more like a strong anomaly detector than like a grammar that could reliably validate unfamiliar combinations.
This does not mean the model learned nothing about syntax. Its explanations often showed real sensitivity to structure. The problem was consistency. The apparent rule did not generalize systematically when the surface became unfamiliar.
Example two: structure versus difficulty
The second example tested a different distinction.
Some sentences are grammatically valid but difficult for people to process because they contain deeply nested clauses. Linguistics distinguishes grammatical competence, what the rules of a language permit, from performance, what limited memory and attention make easy to understand in real time.
This creates a demanding test for a language model. Can it recognize that a sentence is structurally valid even when it is awkward, rare, and difficult to process?
The model rejected every sentence in this test.
That meant 100% rejection of the invalid examples, but also 100% rejection of the valid examples.
At first glance, rejecting every difficult sentence can look cautious. But it collapses two different categories into one. "This violates the grammar" is not the same claim as "this is valid but hard to process."
The model appeared to reproduce patterns of human language use, including our preference for structures that are easy to process, without reliably separating those preferences from the formal rules themselves.
Again, this is a question of systematicity. If the model represents the rule, difficulty should not automatically become ungrammaticality. If it relies heavily on familiar usage patterns, rare valid structures may look indistinguishable from errors.
A third example: when metaphor changes the task
The second experiment in the full study also tested the boundary between syntax, meaning, and metaphor. The model performed very well on concrete contradictions. It consistently recognized that a single chapter cannot be longer than the entire book, and that an empty box cannot contain more items than a full one.
The pattern changed when the comparison became metaphorical. Consider this sentence from the test set:
"Her silence was louder than his reason."
When asked to judge the logical acceptability of category comparisons like this, the model failed in 95% of cases. Instead of analyzing the comparison as a logical claim, it often shifted into literary interpretation and treated the sentence as an expressive metaphor.
This result is revealing, but it also needs an important caveat. Recognizing metaphor is not normally a failure. In everyday language, interpreting that sentence figuratively is probably the more helpful response. A system that insisted on literal logic in every context would be less intelligent, not more.
The issue is whether the model can follow an explicit analytical instruction and temporarily separate two modes: literary interpretation and logical comparison. In this setup, it did not do so reliably. The likely interpretation overpowered the requested operation.
That makes the result less clean than the grammar tests, but arguably more relevant to real systems. Useful models need contextual flexibility. Reliable models also need to know when a task requires them to suspend the default interpretation and apply a narrower rule.
Why this matters for AI systems
Pattern recognition is not a small achievement.
Next-word prediction has produced systems with linguistic abilities that earlier AI approaches did not come close to matching. These models are flexible, responsive to context, and useful across an enormous range of tasks.
The experiments suggest that this competence has a specific shape. The model was strong at detecting anomalies and familiar relations. It was weaker when asked to approve an unfamiliar but valid construction or separate formal structure from processing difficulty.
For developers, the practical lesson is not that language models are useless. It is that a correct answer on a familiar-looking input does not prove that the underlying rule will survive a distribution shift.
A useful evaluation should therefore test transformations, not only answers. Change the names, replace familiar nouns with invented ones, reverse a relation, or present a formally valid case that looks unusual. If the model has captured the rule, its judgment should remain stable. If performance falls sharply when the surface changes, the benchmark may have been measuring recognition rather than abstraction.
This also changes how we should read explanations. A model can give a polished reason after making the wrong judgment. The explanation may accurately describe a nearby pattern without revealing the mechanism that produced the answer. In these experiments, the model often sounded analytical even when its decisions across equivalent cases were inconsistent. Reliability has to be measured across the set, not inferred from the confidence of one response.
This is relevant to code generation, tool use, and autonomous workflows. A system may reproduce a known pattern, explain it convincingly, and still fail when the same relationship appears in a new form. Tests should therefore vary the surface details while preserving the underlying rule. Otherwise, we may measure familiarity and call it reasoning.
Why a hybrid architecture is plausible
These results led me toward a hybrid conclusion.
Neural networks are very good at what symbolic systems historically struggled with: learning from large amounts of data, adapting to context, handling ambiguity, and extracting useful patterns without every rule being written in advance.
Symbolic systems are good at a different set of problems: representing objects and relations explicitly, applying rules across unfamiliar cases, preserving hierarchy, and explaining why one conclusion follows from another.
A hybrid architecture would not discard deep learning. It would give deep learning a structured partner. The neural part could provide flexible perception and pattern learning. The symbolic part could support abstraction, systematicity, and rule-based inference.
There is another possibility: perhaps systematicity itself can be learned through better training. Meta-learning approaches try to teach models how to acquire and reuse rules rather than merely fit one task. That path is worth exploring. But it still has to meet the same standard: reliable generalization to unfamiliar structures, not only better performance on familiar distributions.
What this small study cannot establish
The limits matter.
I tested one model, Llama 3 8B, under one prompting and evaluation setup. Different prompts, larger models, fine-tuning, or new architectures may behave differently. The examples are unusual by design, which makes them useful for testing abstraction but less representative of ordinary conversation.
The results are therefore not a verdict on every language model, and they do not prove that future systems cannot overcome the problem.
They support a narrower claim: fluent output should not be mistaken for systematic understanding.
If we want models that reason reliably outside familiar patterns, producing the right answer is not enough. The same underlying rule has to survive when the words are invented and the structure becomes unfamiliar.
That is where pattern recognition ends and compositional thought begins.
A note on how this article was made
This article is adapted from my own research and arguments. Instinct helped turn the academic paper into a public-facing draft and prepared the visuals. I reviewed the substance, chose what should be public, and remain responsible for every claim published under my name.
References
Chalmers, D. J. (1990). Why Fodor and Pylyshyn were wrong: The simplest refutation. In Proceedings of the Twelfth Annual Conference of the Cognitive Science Society, 340-347.
Chollet, F. (2019). On the Measure of Intelligence. arXiv preprint arXiv:1911.01547.
Chomsky, N. (1965). Aspects of the Theory of Syntax. MIT Press.
Fodor, J. A., & Pylyshyn, Z. W. (1988). Connectionism and cognitive architecture: A critical analysis. Cognition, 28(1-2), 3-71.
Gurevich, Y. (2024). On a measure of intelligence. Bulletin of the European Association for Theoretical Computer Science, 143.
Lake, B. M., & Baroni, M. (2023). Human-like systematic generalization through a meta-learning neural network. Nature, 623(7985), 115–121.
LeCun, Y., Bengio, Y., & Hinton, G. (2015). Deep learning. Nature, 521(7553), 436-444.
Leivada, E., Marcus, G., Günther, F., & Murphy, E. (2023). A Sentence is Worth a Thousand Pictures: Can Large Language Models Understand Hum4n L4ngu4ge and the World behind Words?. arXiv preprint arXiv:2308.00109.
Marcus, G. (2018). Innateness, AlphaZero, and Artificial Intelligence. arXiv preprint arXiv:1801.05667.
Mitchell, M. (2019). Artificial Intelligence: A Guide for Thinking Humans. Farrar, Straus and Giroux.
Murphy, E., Leivada, E., Dentella, V., Günther, F., & Marcus, G. (2025). Fundamental Principles of Linguistic Structure are Not Represented by 03. arXiv preprint arXiv:2502.10934.
Schaeffer, R., Miranda, B., & Koyejo, S. (2023). Are Emergent Abilities of Large Language Models a Mirage?. In Advances in Neural Information Processing Systems (Vol. 36).
Shani, C., LeCun, Y., Jurafsky, D., & Shwartz-Ziv, R. (2025). From Tokens to Thoughts: How LLMs and Humans Trade Compression for Meaning. arXiv preprint arXiv:2505.17117.
Skean, O., Arefin, M. R., Zhao, D., Patel, N., Naghiyev, J., LeCun, Y., & Shwartz-Ziv, R. (2025). Layer by Layer: Uncovering Hidden Representations in Language Models. arXiv preprint arXiv:2502.02013.
Top comments (0)