A protein language model is not a small AlphaFold. It is a masked-token model over amino-acid strings whose output is a vector per residue, and the useful question about it is what those vectors are a function of.
The training objective
The setup is the one BERT introduced for text, applied to a twenty-letter alphabet. Take a protein sequence, replace a fraction of its residues with a mask token, and train the network to recover the originals from the residues that remain. There is no label, no structure, no annotation and no alignment in the loss — the only supervision is the sequence predicting itself.
That objective is why the resulting representation is interesting. To fill in a masked residue well, the model has to learn which positions in a protein constrain which other positions, and in real proteins those constraints are overwhelmingly structural and functional: two residues that touch in the folded protein co-vary across evolution because a change in one is only tolerated alongside a compensating change in the other. A model that gets good at the fill-in task has, as a side effect, absorbed a great deal of that co-variation.
The exact masking fraction and the corruption scheme (how often a masked position is replaced by a random residue rather than the mask token) differ between model generations and are stated in each paper’s methods section. Read the fraction off the paper for the checkpoint you are using rather than assuming the text default carried over.
What the training set actually is
This is the part most explanations get vague about, and it is checkable. For the ESM family from Meta AI, the model cards published in the project’s own repository name the corpus directly: ESM-1b was trained on UniRef50 release 2018_03, and the ESM-2 checkpoints on UniRef50 release 2021_04, with sequences sampled from the UniRef90 clusters those UniRef50 representatives head. The checkpoint names carry it: esm2_t33_650M_UR50D ends in UR50D for that reason. Meta AI’s ESM repository publishes the table.
UniRef50 matters as a choice, not just as a name. UniProt builds it by clustering sequences at 50 percent identity and keeping one representative per cluster, which is a deliberate de-duplication: the raw protein databases are enormously redundant, with thousands of near-identical entries for well-studied proteins and a single entry for something sequenced once from a soil sample. Training on the raw set would weight the model toward whatever humans have sequenced most. Clustering at 50 percent flattens that. Sampling training examples from the UniRef90 members within each UniRef50 cluster then restores some within-family variation without restoring the popularity bias.
The consequence for you is a coverage statement rather than a capability statement. The model has seen the sequence space that has been deposited in public databases, which is dominated by bacterial and archaeal genomes and by well-studied eukaryotic proteomes. Antibody variable regions, intrinsically disordered proteins and designed sequences are all under-represented relative to how often people want to run models on them.
What the embedding encodes
The natural output is one vector per residue, taken from the final transformer layer — for esm2_t33_650M_UR50D that is 33 layers producing 1,280 dimensions per residue. A 300-residue protein comes back as a 300 by 1,280 matrix. If you want one vector for the whole protein you have to pool, and the standard choice is a mean over the residue positions.
What is in those vectors, based on what the published evaluations probe for:
- Contact information. The attention maps of these models contain patterns that correspond to residue pairs in three-dimensional contact, recoverable with a small supervised probe. This is the most direct evidence that the fill-in objective is picking up structure.
- Family and homology. Mean-pooled embeddings of proteins from the same family land near each other, which is what makes embedding-based homology search work at identities where sequence alignment is losing signal.
- Positional tolerance to substitution. The model’s own predicted distribution at a position is a measure of how constrained that position is. Scoring a variant by the log-ratio of the mutant residue’s probability to the wild-type residue’s — a pseudo-likelihood — is the standard zero-shot variant-effect method and needs no labelled data at all.
What is not in them is anything the sequence does not determine. Post- translational modifications, expression level, the presence of a binding partner, the buffer conditions: none of that is an input, so none of it is in the output.
Model size and what it buys
The published ESM-2 checkpoints run from 8 million parameters at 6 layers and 320 dimensions up to 15 billion parameters at 48 layers and 5,120 dimensions, with the 650-million-parameter, 33-layer, 1,280-dimension model the one most work actually uses. That is not conservatism. The embedding dimension is the width of every downstream feature vector you build, and a 5,120-dimension representation of a few hundred thousand proteins is a large matrix to store and index — see what dimension count costs downstream.
The general finding across the scaling work is that the fill-in loss improves smoothly with size and that structural information in the representations improves with it, but the relationship between the two is not linear and the benefit on a specific downstream task is not guaranteed to survive to your dataset. The honest procedure is to run the smallest checkpoint and one larger one on your own labelled set and compare, which is cheap because the head is small; that is worked through in fine-tuning on a small labelled set.
What it does not tell you
The most important distinction is between the language model and the structure predictor. A folding head trained on experimental structures can be attached on top of these representations, and that combined system predicts coordinates; the language model on its own does not. When a paper reports structure-prediction accuracy for a system in this family, it is reporting on the head plus the trunk, trained with structural supervision. Embeddings from the trunk alone are a different product, useful for search, classification and variant scoring.
Second, a pseudo-likelihood score is a statement about how unusual a sequence is relative to the training distribution, not a statement about a phenotype. Variants that these scores rank as tolerated include real disease-causing variants, and variants they rank as disruptive include ones with no measured effect. These scores are used as a prior in research settings and as an input to prioritisation. They are not a clinical determination, and interpreting a human variant for a clinical purpose is a regulated activity carried out against published criteria by an accredited laboratory. A model score does not substitute for it and should not be presented as if it might.
Third, nothing in the objective rewards novelty. A sequence the model scores as very likely is a sequence that looks like the training distribution, which for design work is often the opposite of what you want.
Top comments (0)