Current implementation: empirical z-free-relative reconstruction rescaling

Back to implementation losses

Methodology note: Distribution-Level Z-Free-Relative Reconstruction Rescaling

This note records how the current code implements distribution-level z-free-relative reconstruction rescaling.

What the code does

The current implementation uses one scalar

$$c_{\mathrm{base}}$$

stored as zfree_reference_scale, and defines the distribution-normalized reconstruction term as

$$L_{\mathrm{dist}} = \frac{L_{\mathrm{raw}}}{c_{\mathrm{base}}}.$$

In code, this scalar is used directly to construct the component named:

• distribution_normalized_reconstruction_loss

The raw reconstruction term remains separately available as:

• reconstruction_loss

Current scope

The current implementation uses zfree_reference_scale in exactly two reconstruction terms:

• distribution_normalized_reconstruction_loss
• normalized_concentration_corrected_token_weighted_reconstruction_loss

It does not rescale:

• example_inverse_surprisal_weighted_reconstruction_loss
• token_weighted_reconstruction_loss
• any KL term
• any other auxiliary term

So the current design is still selective rather than a matched normalization of every reconstruction-family member.

Where the scalar comes from

The trainer computes zfree_reference_scale before training in:

• path:experiments/synthetic_sequences/disentangled/train_disentangled.py

Current behavior:

• if the model source is an entangled run, load the saved entangled validation loss
• otherwise, score the current validation records with the pretrained source model and take the mean y-region loss

Configuration surface

The current configuration contract lives in:

• path:core/disentanglement/config.py

under:

• ReconstructionLossConfig.zfree_reference_scale
• ReconstructionLossConfig.distribution_normalized_weight

The current CLI surface is:

• --distribution-normalized-reconstruction-weight

Where the term is assembled

The scalar reconstruction component is formed in:

• path:core/disentanglement/shared_discrete_cvae.py
• path:core/disentanglement/continuous_cvae.py

Both implementations use the same pattern:

$$L_{\mathrm{dist}} = L_{\mathrm{raw}} / \texttt{zfree\_reference\_scale}.$$

Related notes

• Current reconstruction loss stack
• Current implementation: normalized concentration-corrected token-weighted reconstruction