DEV Community

Cover image for The Mathematics of Misreading a Poem
Ipshita Bhardwaj
Ipshita Bhardwaj

Posted on

The Mathematics of Misreading a Poem

There is a particularly interesting inconvenience in using your own poetry as a machine learning dataset: you know what the words were supposed to mean.

Whether they actually convey that meaning is a different question entirely.

I recently decided to turn twelve of my poems into numerical vectors and investigate what happened to the relationships between them. It began as a fairly innocent exercise in curiosity and ended up making me reconsider how I evaluate semantic retrieval systems.

Somewhere in between, three embedding models and I had a disagreement about a mountain.

Faces are only geometry

In one of my poems, I wrote about a painter who had become disillusioned with painting reality.

people were easy.

faces are only geometry.

light resting temporarily on bone.

he wanted to paint the invisible movements beneath them.

The painter's frustration was that he could reproduce someone's appearance with remarkable precision and still feel that he had captured remarkably little of them.

It occurred to me that this was an interesting problem to revisit from a computational perspective.

Text embeddings are, after all, an attempt to make language geometrically meaningful. We take something as structurally and semantically complicated as human language, represent it as a high-dimensional vector, and use mathematical operations to reason about its relationships with other pieces of language.

It is an extraordinarily useful abstraction. Much of modern semantic search and retrieval-augmented generation depends on it.

But poetry is an interesting thing to subject to that abstraction because its meaning frequently depends on the distance between what is said and what is meant.

A mountain need not be a mountain. An empty room might be about someone who is no longer in it. A poem about a child made of wax might have very little to do with wax.

What would the geometry of such a collection look like?

More importantly, what would determine it?

Constructing a small literary universe

I assembled a corpus of twelve original poems, varying considerably in length, subject matter and writing style.

Some were about resilience and personal growth. Others explored grief, perception and emotional reality. Two were alternative narratives involving Rue from The Hunger Games.

I generated embeddings using three pretrained models:

  • all-MiniLM-L6-v2 — 384 dimensions
  • all-mpnet-base-v2 — 768 dimensions
  • BAAI/bge-base-en-v1.5 — 768 dimensions

I also included TF-IDF as a lexical baseline, a decision that became considerably more important later in the experiment.

There was an immediate implementation problem. Some of my poems exceeded the models' maximum input lengths.

Truncation is a fairly ordinary concern in NLP pipelines, but there is something especially unfortunate about silently removing the ending of a poem. Sometimes the ending is precisely what establishes the meaning of everything preceding it.

I therefore split longer poems into token-based chunks, embedded each individually, and aggregated them using a token-weighted average:

ep=∑i=1ntiei∑i=1nti e_p = \frac{\sum_{i=1}^{n} t_i e_i}{\sum_{i=1}^{n} t_i}

Here, e_i is the normalised embedding of a chunk and t_i is its token count. The resulting poem-level embedding was then normalised again.

This preserves contributions from the complete text, although it introduces its own compromise: averaging representations does not explicitly preserve the order in which their constituent ideas appear.

That distinction is particularly relevant to poetry. Reordering twelve lines might leave their vocabulary largely unchanged while producing a very different poem.

With the representations ready, I calculated pairwise cosine similarities and used PCA to explore the resulting space visually.

Pairwise cosine similarity heatmap of twelve original poems using MPNet embeddings
Figure 1: Pairwise cosine similarity across my twelve poems using MPNet embeddings.

Two-dimensional PCA projection of twelve poems in MPNet embedding space
Figure 2: Two-dimensional PCA projection of the same poetry embeddings.

The two-dimensional projection is useful for exploration, although it necessarily distorts some of the relationships present in the original embedding space.

Twelve poems. Sixty-six unique relationships per model. Several hundred dimensions in which to locate things I had once struggled to put into words.

The geography of resemblance

The most consistent relationship in the corpus was also the least surprising.

Two poems, and the canary kept singing and the orchard fed her bones, reimagine Rue's fate in The Hunger Games. They share characters, a fictional universe and several narrative elements.

Every embedding model ranked them as the most similar pair.

A considerably more interesting relationship emerged elsewhere.

In wax child, I wrote about a mother grieving a child who was made of wax.

they said

“he was never real.”

but what do they know

of the way we feel?

I associate this poem with the one about the painter. Both explore the inadequacy of physical reality as a measure of emotional truth.

Their narrative settings, vocabulary and imagery, however, are quite different.

Here is where the models placed that relationship among the 66 possible poem pairs:

Model Cosine similarity Relative rank
MiniLM 0.542 6th
MPNet 0.572 12th
BGE 0.585 53rd

Notice something slightly counterintuitive.

BGE produced the highest cosine similarity of the three, yet regarded the poems as comparatively less similar within its own representation of the collection.

This is why interpreting raw similarity scores across independently trained embedding spaces can be misleading. A similarity of 0.58 in one space need not carry the same significance as 0.58 in another.

I do not consider MiniLM's ranking inherently more correct simply because it resembles my own interpretation. Authorial intent is one way of establishing a relationship between poems, not an exhaustive account of their possible relationships.

Still, the disagreement made me curious about a more specific question.

Could the models distinguish an intended metaphorical meaning from a superficially similar literal one?

A computational disagreement about mountains

Consider these two lines from another of my poems:

these mountains that you are holding,

you were only supposed to climb.

The distinction between holding and climbing is essentially the premise of the poem. It concerns the tendency to turn temporary difficulties into permanent burdens.

I compared the excerpt against two candidate interpretations.

Figurative:

You are treating your difficulties as permanent burdens rather than challenges to overcome.

Literal:

These mountains are difficult to climb while holding heavy equipment.

The models had a rather different opinion of which sentence resembled my writing.

Model Figurative similarity Literal similarity
MiniLM 0.272 0.622
MPNet 0.265 0.576
BGE 0.504 0.669

Apparently, I had written something closely related to mountaineering.

I repeated the experiment with five other excerpts.

One of them was from Broken Glass:

broken glass reflects more light

than something whole ever could.

MPNet assigned a similarity of 0.126 to a figurative interpretation concerning transformation through hardship, and 0.823 to a literal description of the optical properties of broken glass.

The literal alternatives were preferred in every comparison.

Six excerpts. Three models. Eighteen comparisons. Not a single figurative win.

Figurative-minus-literal cosine similarity margins across six poetry excerpts using MiniLM, MPNet and BGE
Figure 3: Figurative-minus-literal cosine similarity margins across six poetry excerpts and three embedding models. Negative margins indicate a preference for the literal alternative.

It was an almost suspiciously elegant result.

And elegant results can be dangerous when you are particularly fond of the hypothesis they appear to support.

The experiment had a metaphor of its own

I had designed an evaluation in which the intended interpretation and the alternative differed along more than one dimension.

The figurative candidates preserved the intended meaning but frequently discarded the original vocabulary. The literal candidates preserved much of that vocabulary while changing the intended meaning.

Consequently, I could not easily distinguish a preference for literal interpretation from a preference for lexical resemblance.

There was also a contextual asymmetry.

I possessed the complete poem, its intended meaning and the circumstances of its composition. The models received two lines.

I was comparing representations produced under radically different informational conditions.

Even a human reader could legitimately interpret an isolated poetic phrase differently from its author. Why should I consider that ambiguity irrelevant when constructing a computational evaluation?

The problem was no longer simply whether an embedding model could represent metaphor. It was whether my experiment could distinguish metaphorical representation from other sources of similarity.

Those are different questions.

So I changed the experiment.

Giving the mountain its surroundings

The follow-up consisted of 288 similarity calculations involving the original six excerpts, their complete poems, multiple candidate formulations, three embedding models and the TF-IDF baseline.

I introduced alternatives that retained some of the metaphor's original vocabulary while preserving its intended figurative meaning.

For example:

The mountains you carry are challenges to climb, not lifelong burdens.

This is much closer to the surface form of the original excerpt than the initial figurative paraphrase, without reducing the mountains to geographical objects.

I also repeated the comparisons using complete poems rather than isolated excerpts.

Instead of examining a similarity score independently, I considered the difference between the two candidate scores:

Δ=cos⁡(q,f)−cos⁡(q,l) \Delta = \cos(q, f) - \cos(q, l)

Here, q represents the poem or excerpt, f the figurative candidate and l the corresponding literal candidate.

A positive margin means the figurative candidate ranks higher.

The results across the three embedding models looked like this:

Candidate formulation Excerpt Full poem
Original alternatives 0/18 6/18
Alternative low-overlap wording 8/18 14/18
Approximately vocabulary-matched wording 10/18 16/18

Comparison of isolated excerpts and full-poem context using the original figurative and literal candidates
Figure 4: Effect of full-poem context using the original candidate interpretations.

Effect of full-poem context on figurative similarity margins using alternative low-overlap candidate formulations
Figure 5: Effect of context using alternative low-overlap candidate formulations.

Comparison of figurative similarity margins between isolated excerpts and complete poems under approximately vocabulary-matched conditions
Figure 6: Effect of context under approximately vocabulary-matched conditions.

My wonderfully straightforward result had become considerably less straightforward.

Under the approximately vocabulary-matched, full-poem condition, MPNet and BGE preferred the figurative candidates in all six examples. MiniLM did so in four.

More interestingly, MPNet's preference for the literal interpretation of the mountain excerpt reversed when I provided the complete poem, without changing the original candidate interpretations.

The same model. The same candidate sentences. A different amount of context. A different ranking.

Of course, changing the candidate wording and changing the context simultaneously does not allow me to attribute the overall improvement to either factor independently. The separate conditions are useful precisely because they let us examine those changes individually.

And then there was TF-IDF.

Despite lacking contextual semantic representations, the lexical baseline also improved under several of the modified conditions. Given complete poems and alternative low-overlap candidates, it preferred the figurative interpretations in all six examples.

That complicates the more appealing explanation that additional context simply allowed the neural models to discover the deeper meaning.

Sometimes additional context also means additional relevant words.

The uncomfortable familiarity of the problem

At this point, the experiment started reminding me less of literary interpretation and more of retrieval engineering.

Suppose a user asks a question in a RAG system using terminology that differs significantly from the relevant document. Meanwhile, an irrelevant document happens to contain several of the same words as the query.

Or suppose the information required to interpret a passage is distributed across two chunks, only one of which gets retrieved.

Or suppose an evaluation dataset contains negative examples that are systematically more lexically similar to the queries than its positive examples.

These are variations of the problems I encountered while experimenting with poetry.

The application is different. The underlying representation and evaluation problems are remarkably familiar.

We often think of chunk size as an implementation parameter, something to adjust according to token budgets and retrieval latency. But it also determines how much of a thought survives as a coherent retrieval unit.

Similarly, candidate formulation can become an unintended source of bias in an evaluation. A model might appear to succeed or fail because of how the alternatives were constructed rather than because of the specific capability we intended to measure.

And a simple baseline can reveal that a supposedly sophisticated improvement is at least partly attributable to something much more ordinary.

There is a tendency to move rather quickly from observing model behaviour to explaining model behaviour.

My first experiment was a useful demonstration of why those steps deserve to remain separate.

What the experiment cannot tell me

Twelve poems constitute a very small literary universe.

Six selected metaphors cannot establish a general conclusion about figurative language comprehension. My candidate interpretations and thematic annotations were prepared with AI assistance and were not independently validated by multiple human annotators. The code was also developed with AI assistance.

The vocabulary controls were approximate rather than exact. The models differ in their training objectives, and chunk-level averaging introduces further representational compromises.

Most importantly, cosine similarity measures proximity between representations. It does not directly measure comprehension.

What I can say is that the retrieval rankings in this corpus were sensitive to context and candidate formulation, and that the initial preference for literal interpretations was substantially reduced under modified experimental conditions.

The distinction might seem unnecessarily cautious for a personal project.

But I would rather have a small experiment that answers a carefully defined question than an impressive-looking experiment that answers a question I never actually tested.

What the painter couldn't paint

Towards the end of my poem about the painter, he finally produces a painting he is satisfied with.

It is an empty room.

because for the first time in years,

he had not painted what something looked like.

he had painted

what it meant to lose it.

There is something amusing about returning to those lines after spending time calculating the geometric relationships between my poems.

The experiment gave me a new way of examining my writing. I could compare pieces I had never consciously associated with each other, inspect their neighbourhoods in different embedding spaces, and observe how those relationships changed with the model.

None of that requires the vectors to constitute a complete account of the poems. They are useful precisely because they are selective representations constructed for particular computational purposes.

The more interesting engineering question is whether the information retained by a representation is sufficient for the task we assign to it.

I began this project wondering whether an embedding model would misread my poetry.

I ended it having discovered how easily I could misread an embedding model.

The experiment is available on GitHub:

Poetry in Vector Space

The repository contains both experimental notebooks, the poetry corpus, candidate interpretations, similarity scores, visualisations and reproduction instructions.

If you write poetry, or anything that makes particularly unconventional use of language, I would be interested to see what its geometry looks like.

Top comments (0)