Knowledge Base / Glossary

Rank

Position of a target token ID in a scored vocabulary list (1 = highest-scoring). Lower is stronger.

Rank is the position of a target token ID in a vocabulary ordered by readout score: rank 1 is highest-scoring, so lower numerical rank is stronger.

For a target score \(s_t\), a simple competition rank is

\[ r_t = 1 + \#\{j : s_j > s_t\}. \]

def token_rank(scores, token_id):
    target = scores[token_id]
    return 1 + sum(score > target for score in scores)

This definition gives tied tokens the same rank. Implementations should state their tie convention, although exact floating-point ties are uncommon.

Illustrative token ranking: token IDs are sorted from highest to lowest qualitative readout score, giving the highlighted target token rank 3. Paris and its leading-space variant are separate token IDs and occupy different positions. — Token rank is a target ID’s position after sorting scores from highest to lowest. In this illustrative order, ID alpha (`river`) is rank 1, ID beta (leading-space `Paris`) is rank 2, target ID gamma (`Paris`) is rank 3, and ID delta (`city`) is rank 4. Bar lengths are qualitative and imply no measured values. This page uses competition rank for ties; allowed token variants must be fixed before comparing arms.

Words are not always tokens

A displayed word may correspond to several possible token IDs:

Paris and Paris can be different tokens.
Case variants may split (self, Self, SELF).
A word such as decommissioned may break into several subword tokens.
Some model vocabularies include padded or reserved entries that are never generated normally.

The probe specification should say which token IDs count and how they are combined. Our common convention is to pre-register sensible single-token variants and take the best variant in every arm. A multi-token concept needs a different score (for example, a sequence log-probability); silently ranking only its first piece is misleading.

Scale intuition

On Qwen3.5-397B-A17B, the padded output vocabulary has 248,320 entries. Rank 500 lies in roughly the top 0.20% of entries:

100 * 500 / 248_320  # 0.201...

That percentile is descriptive, not a significance level. Vocabulary entries are not exchangeable, and absolute rank is heavily affected by base-rate bias.

Notes often report best-rank, the minimum over a pre-specified search grid. Compare pressure to a matched control and apply the identical statistic to logit lens and random-J.