Affine map
A transformation that applies a linear map and then adds a fixed offset; it preserves …
Attention mask
A model or library rule that marks usable positions or specifies which sequence positions …
Base-rate bias
A low absolute probe-token rank can reflect its prompt-conditioned baseline or a …
Basin of attraction
The set of starting states whose trajectories converge to a given attractor, such as an …
Best-rank
The minimum vocabulary rank a pre-specified probe token hits across layers, positions, and …
BF16, FP16, and FP32
Floating-point formats for model arithmetic: 32-bit floating point (FP32), standard 16-bit …
Bootstrap
A resampling method that repeatedly draws observed units with replacement to study how a …
Brier loss
The average squared gap between a stated probability and the zero-or-one outcome that …
Brouwer's fixed-point theorem
A continuous map from a nonempty compact convex set into itself has at least one fixed …
Causal attention
A rule that lets each token use itself and earlier tokens, and not later ones.
Convex hull
Every weighted average of a set of points whose weights are nonnegative and sum to one.
Cosine similarity
A comparison of two nonzero vectors based on the angle between them rather than their …
Covariance matrix
A square table of how a random vector spreads around its mean in every direction: …
Data leakage
An accidental flow of information across an evaluation boundary that makes held-out …
Dtype
The data type (dtype) of a tensor or array: the storage format shared by every element, …
Embedding
A vector of numbers used to represent an item so a model or retrieval system can compare …
Feed-forward network (FFN)
A computation with no feedback cycle; in a Transformer, usually the shared position-wise …
Fisher's exact test
An exact conditional test for association in a count table, commonly used for a 2 by 2 …
Forward hook
A callback registered on a neural-network layer that runs during each forward pass to …
Forward-mode automatic differentiation
A way to compute a program's derivative along one input direction by carrying a tangent …
Gini impurity
A measure of how mixed the class labels are inside a group of rows, used by decision trees …
Greedy continuation
The model's generated text when each next token is chosen as the single …
Hidden bridge
An intermediate entity that logically links a two-hop question to its answer while …
Hysteresis
A system's response can depend on how its controls arrived at their present values.
i.i.d.
Independent and identically distributed: each draw is independent of the others and from …
Imputation
A documented rule for filling missing values so an analysis can use a complete table …
Jacobian lens
A corpus-averaged linear map that turns a mid-layer residual-stream state into …
Krawczyk's test
An interval Newton test: if Krawczyk's operator maps a box strictly inside itself, the box …
L2 normalization
Rescaling a nonzero vector to Euclidean length one while preserving its direction.
Log loss
The average negative logarithm of the probability a model gave to what actually happened, …
Logit
An unnormalized next-token score. Softmax turns a vector of logits into probabilities.
Logit lens
The free mid-layer readout that skips any fitted transport: unembed the residual as if it …
Lyapunov exponent
The long-run exponential rate at which an infinitesimally small perturbation of a …
Matmul
Matrix multiplication: the core linear-algebra operation behind attention, multilayer …
Mean pooling
A rule that averages selected vectors to produce one summary vector.
Monodromy
The derivative of one full trip around a periodic orbit: the ordered product of the step …
Motor layer / motor-late
A late, near-output regime; in one scoped 397B analysis, digit-direction vocabulary-score …
Multilayer perceptron (MLP)
The position-wise nonlinear feed-forward sublayer that transforms each token state and …
Nat
The natural-logarithm unit of growth and uncertainty: one nat is about 1.44 bits, and an …
Next-token entropy
The Shannon entropy of one next-token probability distribution: zero when one token has …
Oseledets' theorem
If a differentiable map preserves a probability measure and its derivative is not too …
p-value
A tail probability under a stated null model: how often that null produces a result at …
Pearson correlation
A number from negative one to positive one that summarizes the direction and strength of a …
Principal component analysis (PCA)
A linear dimensionality-reduction method that replaces numerical feature columns with new …
Prompt echo
When a lens lights up words that were already in the prompt; looks like a finding, often …
Random-J
A family of randomized Jacobian-lens controls; each construction states which scale or …
Rank
Position of a target token ID in a scored vocabulary list (1 = highest-scoring). Lower is …
Relevance judgments
Reference labels that say which documents are considered relevant to each evaluation …
Residual stream
The model's running hidden-state channel: each layer reads it, adds an update, and passes …
Retrieval ranking metrics
Measures that compare a ranked retrieval list with relevance judgments, including …
RMS normalization
Rescaling a vector by a stabilized root mean square of its coordinates, then by a learned …
Root mean square (RMS)
The square root of the mean squared coordinates of a vector, used here as a measure of …
Rotary embedding
A transformer position method that rotates pairs of attention-query and attention-key …
Sampling
Choosing the next token from a distribution over the vocabulary, rather than always taking …
SiLU
The activation that multiplies a number by its own sigmoid, smooth through zero.
Soft-token feedback
A feedback rule that feeds back a probability-weighted average of token embeddings instead …
Stable manifold
The set of states whose trajectories converge to a given fixed point or periodic orbit; …
Standard deviation
A measure of how far values in one numerical feature typically spread around their mean.
Standard scaling
A feature-wise transformation that subtracts each column's mean and divides by that …
Survival-identity vocabulary
The frozen probe list for the primary self-vs-other-model battery; its exact normalized …
Temperature
A scalar that sharpens or flattens the next-token distribution before sampling.
Thinking on / off
The template-controlled reasoning protocols used in these notes: reasoning enabled, or …
Tokenization
The model-specific process that divides an input into tokens and maps those tokens to …
Top-p
Nucleus sampling: keep only the smallest set of tokens whose cumulative probability …
Transformer
A family of sequence-processing neural networks that combine attention, position-wise …
Unembedding
The final linear map from residual-stream coordinates to a score for every vocabulary …
Wilcoxon signed-rank test
A paired test that uses each gap's direction and its absolute-size rank, not just win …
Workspace
A workspace-like mid-network regime studied through selected residual directions and …