praxagent / Methods
Experiment integrity
Credibility is chronological. A reader should be able to check that the design came before the outcomes, that the run followed the design, that nothing was dropped, and that anything added later carries its date.
01 / Why
Why the order matters
Confirmatory research goes wrong when the rules change after the data arrive: stopping data collection once the result looks good, reporting only the comparisons that worked, or writing the hypothesis to fit the data. Judgment calls cannot be removed from a study. They can be dated, so a reader knows which were made before the outcomes and which after.
A dated record lets a reader make four checks.
- The design was committed before any target outcome existed.
- The run followed that design, and every departure is recorded as a dated amendment.
- Every outcome is still in the record, including failures and runs that produced no data.
- Anything added after the outcomes were seen carries the date it was added.
In one of our studies the follow-up failed its own replay gate, so its endpoint results are labeled exploratory; the example at the end of this page links where that happened.
02 / Sequence
Freeze, run, audit, release
Lock the design before outcomes exist
The question, the sample, the controls, the endpoints, the analysis, and the claim boundary (what the result could and could not be taken to show) are committed before any target outcome is generated or inspected.
Execute the plan as frozen
Do not change the rules because of what the data show. When a defect forces a change, record it in a dated amendment that states what had already been observed at the time.
Check the run against the plan
Runtime records are compared with the frozen plan: same sample, same controls, same endpoints. Every condition and every failure stays in the record, including runs that produced no rows.
Release what the claim rests on
Code, prompts, data, and result records let readers inspect the evidence behind each claim.
03 / Labels
Say what the result is
Four labels mark where a study sits relative to its outcomes. The label records when the design was fixed, and that decides how much weight a result can bear:
Outcomes may still shape design
Useful for discovery and feasibility, and reported as exploratory rather than confirmatory evidence.
The design predates the outcomes
The complete plan was committed before target outcomes existed.
Everything reported was in the freeze
"Frozen" describes a plan; "confirmatory" describes a result. A result is confirmatory only if its endpoint, sample, controls, exclusions, and analysis were all in the freeze and the run met the plan's own pass rules. A frozen study whose run fails a planned gate does not produce confirmatory endpoints.
Added after the results were seen
Sensitivity checks and reanalyses added after the outcomes were seen can be informative. They are filed in a dated amendment that states what had already been observed, and they are never promoted to confirmatory status.
04 / In public
What a reader should be able to check
Run the control.
Matched baselines and null tests named in the plan determine what survives into the conclusion.
Report the failure.
Negative results and broken hypotheses remain part of the public record.
Date the changes.
Every change made after the outcomes were seen sits in a dated amendment that says what was already known. The headline is never quietly rewritten.
These are the standards we hold our own research notes to. If a research note does not link its plan, its results, and its dated amendments, treat its claims as unverified.
A worked example: detecting SAE steering
The steering study asked whether a pinned internal readout could detect a specified SAE steering intervention. An SAE (sparse autoencoder) is a model trained to break a network's internal activations into a sparse set of features; steering adds or subtracts one of those features during generation. The study links a protocol committed before outcomes, a report with results and evidence, and a dated follow-up amendment. Before the follow-up could run any new endpoint, its plan required it to reproduce the earlier study's saved readouts within a tolerance fixed in advance (a maximum absolute error of 0.02 logit units). The replay reached 0.25, so the gate failed and the follow-up's endpoint results are reported as exploratory. A reader can trace that change in status instead of inferring it from the headline.
What this does not guarantee
Freezing a plan fixes its rules; it does not make them good ones. A frozen design can still be one that cannot answer its question. A Git commit is a timestamp, not a registry deposit, so unless a plan was also registered we call it prospectively frozen, not preregistered. Our release audits are automated recomputation; they check arithmetic and provenance, not the design, and they are not independent human review.
Revision, 2 October 2026: reorganized the presentation and corrected wording. The example's status and its links are unchanged.