Reconstruction Weighting Redesign Working Map

This note is the temporary working map for the reconstruction-weighting redesign discussion. It is the coordination layer for the conversation: it records the current agenda, the naming scheme, the main decision points, and the knowledge-base notes touched by each checkpoint.

Companion design note:

Reconstruction loss stack API design

Scope

The immediate goal is to separate:

methodology notes that describe the conceptual objective families,
implementation notes that describe what the current code actually does,
ideation notes that track open redesign directions and decisions in progress.

This working map should remain readable enough to act as a shared agenda for future discussion.

Working naming scheme

The current naming grammar is:

level: dataset-level, example-level, token-level
signal: baseline, surprisal, inverse-surprisal, predictive-entropy
mechanism: normalization, weighting, interpolation

This grammar is meant to make neighboring variants easy to distinguish without relying on overloaded interpretive labels.

Current methodology note names

Reconstruction weighting and normalization: hub note
Distribution-Level Z-Free-Relative Reconstruction Rescaling: current distribution-level method
Example-level inverse-surprisal weighting: under-consideration example-level method
Token-level surprisal-share weighting: current token-level method
Normalized concentration-corrected token-weighted reconstruction: under-consideration token-level normalized variant
Unified reconstruction aggregation ideas: deferred future direction

Current implementation note cluster

Current decisions

Keep methodology and implementation separate.
Use implementation notes to document current code behavior only.
Keep redesign proposals in ideation and methodology until they are adopted.
Use standardized names that expose level, signal, and mechanism explicitly.
Treat the current token-level method as token-level surprisal-share weighting.
Treat the current example-level method as not implemented.
Keep tempering in scope for token-level weighting only.
Do not introduce denominator tempering into example-level inverse-surprisal weighting; preserve its fractional interpretation as a separate objective family.
For the next redesign pass, keep reconstruction aggregation as a simple linear combination of explicit reconstruction losses.
Track all five reconstruction losses during training regardless of whether their active coefficients are zero:
- unnormalized reconstruction loss,
- distribution-level z-free-relative reconstruction loss,
- example-level inverse-surprisal weighted reconstruction loss,
- token-level tempered surprisal-share weighted reconstruction loss,
- normalized concentration-corrected token-weighted reconstruction loss.
Keep token-level tempering as the only active tempering knob, with default $\alpha = 1$ .
Defer unified multi-level reconstruction aggregation until the simpler linear-combination redesign has been implemented and evaluated.
Treat normalized concentration-corrected token-weighted reconstruction as the fifth explicit reconstruction term in the future linear-combination stack.
Expose raw token-weighted reconstruction and normalized concentration-corrected token-weighted reconstruction as separate weighted objective terms rather than a token-level kind switch.

Checkpoints

Checkpoint 1: taxonomy and current-state documentation

Create a temporary working map in ideation
Create an implementation/losses note cluster for current behavior
Normalize the primary names of the current methodology notes
Review the checkpoint before moving deeper into redesign

Checkpoint 2: tempered weighting

Discuss token-level surprisal tempering
Exclude example-level inverse-surprisal tempering from the current redesign scope
Generalize the concentration-correction idea from $\alpha = 1$ to general $\alpha$
Decide whether token-level tempering should be framed as a core methodological family
Update methodology notes accordingly

Checkpoint 3: surprisal vs predictive entropy

Compare token-level surprisal weighting against predictive-entropy weighting
Clarify what each signal measures and what assumptions it makes
Record the comparison in methodology notes

Checkpoint 4: composition and redesign

Formalize the explicit linear-combination reconstruction stack for the next implementation pass
Record the proposed config/API for the five-term reconstruction stack
Record implementation design principles for tracking all five reconstruction variants during training
Keep unified weighted reconstruction families in a deferred note rather than adopting one now

Open questions

What is the cleanest parameterization of tempering for token-level weighting?
If example-level weighting later needs an additional emphasis control, how should that be parameterized without breaking the fractional interpretation?
Should predictive entropy be treated as the main conceptual alternative to realized-token surprisal?
How should the future linear-combination stack expose redundant reconstruction variants without making the config unnecessarily confusing?
When unified aggregation is revisited later, which parts of the current explicit stack should collapse into one configurable reconstruction family?
What diagnostics should the redesigned implementation log so that weighting behavior is inspectable?

Checkpoint log

2026-04-21: checkpoint 1 started. Created the working map, began the implementation loss-note cluster, and normalized the primary names of the current methodology notes.
2026-04-21: checkpoint 1 reviewed after link validation. Checkpoint 2 is now open.
2026-04-22: checkpoint 2 scope narrowed. Tempering remains in scope for token-level weighting only; example-level inverse-surprisal weighting remains untempered so that its fractional interpretation stays intact.
2026-04-22: composition decision updated. The next redesign pass will keep a simple linear combination of four explicit reconstruction losses and defer unified reconstruction aggregation to a separate future-looking note.
2026-04-22: added a new candidate token-level objective: normalized concentration-corrected token-weighted reconstruction. This generalizes the $\alpha = 1$ concentration factor to arbitrary token-weight tempering and uses the same outer $c_\mathrm{base}$ normalizer as distribution-level z-free-relative reconstruction.
2026-04-23: token-level stack decision updated. The future linear-combination reconstruction stack should expose raw token-weighted reconstruction and normalized concentration-corrected token-weighted reconstruction as separate weighted terms, bringing the planned stack to five reconstruction losses in total.
2026-04-23: added a concrete future API proposal for the five-term reconstruction stack, including config field names, CLI flags, and approved scalar diagnostics.