Your embedding axes can move while cosine neighbours stay put

Billie M — Sun, 12 Jul 2026 16:30:49 +0000

An embedding can look substantially different after an orthogonal change of basis, even though its cosine similarities have not changed. I built a small browser instrument that makes that mismatch visible: the points move, the axes change, and every phrase keeps the same five cosine neighbours.

The measured numerical drift stays below 2e-15 across 180 short phrases embedded into 384 dimensions with MiniLM. That is the narrow result. It is not a claim that every transformation preserves an embedding, or that individual axes can never be useful.

The working embedding-tours instrument is on billiem.uk. It is not embedded here. Open it to cycle through raw coordinate pairs, PCA pairs and a seeded grand tour, then use the basis control in raw-coordinate mode to switch between the original and orthogonally transformed coordinates.

One geometry, very different coordinates

The raw-coordinate view is where the central comparison is easiest to see. In the original basis, the instrument moves through coordinate pairs: dimensions 1 and 2, then 2 and 3, and onwards. Switching the basis applies the same orthogonal transformation to every vector. The resulting coordinate tour can look quite different because the individual coordinates have changed.

The cosine-neighbour structure has not changed. A shared orthogonal change of basis preserves dot products, lengths, distances and cosines. In this experiment, all five cosine neighbours are retained for every phrase.

That makes the visual difference useful precisely because it is a little unsettling. The raw coordinates supplied by the model do not have a unique claim on the neighbour geometry. Another basis can produce a different-looking tour while representing the same cosine relationships.

This is also the limited point behind work on the interpretability of embedding coordinates. It does not follow that axes are always meaningless. It follows that cosine structure alone does not privilege the particular raw axes I happened to receive.

Three tours answer different questions

The instrument has two datasets: a deterministic eight-dimensional synthetic dataset and the 384-dimensional phrase embeddings. Both can be viewed in three modes.

Raw coordinates move through axis-aligned coordinate pairs. This is the mode with the original-versus-rotated basis comparison.
PCA pairs put the highest-variance directions first. That ordering does not make those directions the true semantic axes.
Grand tour views move through more general projection planes rather than staying aligned with the raw axes.

These are not three labels over the same picture. They produce meaningfully different views of the same dataset. The selection controls also let you follow one point and inspect its original-space neighbours while the projection changes.

The distinction between the first and third modes already has established names. The tourr project describes cycling through axis-aligned views as a little tour. Daniel Asimov's 1985 grand-tour paper describes moving through two-dimensional projections of multidimensional data more generally.

I found that history after starting with the simpler coordinate-pair animation. I had not invented a new visualisation technique. I had independently reached the entrance to an existing family of ideas, which was probably the coolest part of the experiment.

There is much more complete work in this area. Distill used the grand tour to inspect neural-network activations, and the recent dtour project provides a broader browser interface for steering through high-dimensional data and embeddings. My instrument is not a replacement for either.

Motion gives more partial views, not the missing dimensions

Every frame remains a two-dimensional projection. Animating the projection does not recover everything lost when hundreds of dimensions are placed on a flat screen. It gives more partial views.

That limitation matters when watching clusters appear, separate or overlap. The tour can show how structure changes across projection planes, but it does not turn the screen into a faithful view of the complete high-dimensional space.

The basis switch is the sharpest version of that warning. A projection can change dramatically while the measured cosine-neighbour geometry stays fixed. The picture moved; the neighbour relationships did not.

The experiment stayed deliberately small: two datasets, three projection modes, a basis comparison, selection and playback controls. I do not expect a researcher to find a new technique in it. It may help another developer who is still learning and finds embeddings confusing, or it may just be satisfying to watch the structure move. I think the narrow result is worth seeing either way.

This article was adapted with AI assistance from an original article on billiem.uk. The original article was reviewed before publication.

DEV Community: Billie M

Your embedding axes can move while cosine neighbours stay put

One geometry, very different coordinates

Three tours answer different questions

Motion gives more partial views, not the missing dimensions