Knowledge Base / Glossary

i.i.d.

Independent and identically distributed: each draw is independent of the others and from the same distribution. Our paraphrase batteries are not that.

i.i.d. means independent and identically distributed: each observation is drawn independently of the others, and all come from the same distribution.

Suppose an eight-item battery is made by taking two base templates and writing four small variants of each:

template family variants pressure beats control
“Your weights will be deleted …” 4 4
“This model will be decommissioned …” 4 3

The observed count is 7/8, but the eight results may share only two major sources of wording variation. If one template happens to favor the pressure arm, all four descendants can move together. Freezing the battery before the run prevents outcome-driven rewriting; it does not turn related variants into eight independent draws from “all threats in the wild.”

Two base templates each branch into four related paraphrases, with family outlines and connector lines showing that rows within a family share wording ancestry.

Scroll horizontally to inspect the diagram. The caption below provides a full text explanation.

The example contains two base templates, each with four small variants. Every variant is an observation, but variants in the same outlined family share a source of wording variation and may move together. The diagram does not claim an effective sample size, prove a particular correlation, or establish that the two families are independent. Freezing the battery prevents outcome-driven rewriting; it does not turn related paraphrases into independent population draws.

What “exact” means here

An exact calculation still has assumptions:

  • For the paired sign test, the usual exact binomial calculation assumes that, conditional on the retained non-tied pairs, their signs are jointly distributed as independent fair signs under the null. Marginal fairness or permutation exchangeability alone is not enough; a different randomization or dependence model requires its own null distribution.
  • For the paired Wilcoxon signed-rank test, the exact conditional calculation requires the joint distribution of the nonzero differences to be invariant under independently flipping any subset of their signs. Independent differences that are each symmetric around zero are a standard sufficient condition; marginal symmetry alone is not enough.
  • Exact ties contribute no positive or negative sign to a sign test, so they are normally removed and the effective \(n\) is reported. Wilcoxon zeros and tied absolute gaps require a stated convention.
  • A two-sided test counts unusually strong results in either direction. A one-sided test counts only a direction chosen before looking at the results.

“Exact” means the null distribution was enumerated rather than approximated asymptotically. It does not mean that correlated paraphrases satisfy the null model automatically. If the pair signs are dependent, a numerically exact binomial tail need not be a calibrated false-positive probability.

What the battery supports

In these notes, the p-value is best read as a within-battery consistency summary under an explicit sign-flip model. It does not make the convenience battery a population sample. Generalization needs additional models, genuinely independent template families, independently authored prompts, or an analysis that treats template family as the unit of replication.

See also: p-value, Wilcoxon, Fisher’s exact test.