Apr24 Best Loss Attention Pattern
Apr24 Best Loss Attention Pattern
Back to results · Best-loss dynamics
This is an observational attention-pattern replay for the best Apr24 loss setting, dist_plus_tokcc_a0p5_b0p01_d50_lin50. To mirror the Apr22 attention-pattern analysis, this page uses the full adaptation run for each task at the selected loss setting.
Scope
The replay generated 1,536 traces: 256 sampled test examples for each of the six tasks. Of these, 1,535 traces had analyzable answer tokens, yielding 415,448 token-layer rows.
For each trace, the analysis samples z from the router, replays the conditioned generation with attention enabled, and compares per-token attention to the predictive z placeholder against the token-level negative log likelihood under a z-free reference model. The soft attention summaries use max-over-head attention to the predictive z position. The hard summaries measure whether any head's top attended key is the predictive z position.
Example Traces
The example panels show selected traces by task with token-level base-model loss and predictive-z attention. These panels are intended as qualitative checks on whether high-loss answer tokens line up with attention to the latent z placeholder.
The hard-attention version shows the same examples using top-key hits rather than soft attention mass.
Layer-Level Alignment
Across all tasks, every layer has positive bootstrap confidence intervals for Spearman alignment between base-token loss and predictive-z attention. The strongest overall Spearman alignment is in zero-indexed layer 7 at 0.2712, while the strongest Pearson alignment is in layer 4 at 0.2811.
| Layer | Spearman loss-attention alignment | Pearson loss-attention alignment | Mean predictive-z attention | Hard top-key fraction |
|---|---|---|---|---|
| 0 | 0.2614 | 0.2540 | 0.0825 | 0.1370 |
| 1 | 0.1760 | 0.2446 | 0.0916 | 0.2651 |
| 2 | 0.1977 | 0.2232 | 0.1128 | 0.3476 |
| 3 | 0.2349 | 0.2076 | 0.1438 | 0.4338 |
| 4 | 0.2396 | 0.2811 | 0.1127 | 0.4093 |
| 5 | 0.2346 | 0.2628 | 0.0929 | 0.3879 |
| 6 | 0.2546 | 0.2506 | 0.0820 | 0.3734 |
| 7 | 0.2712 | 0.2503 | 0.0852 | 0.3592 |
The hard top-key baseline from random key selection is 0.0350. The observed hard top-key fraction peaks in layer 3 at 0.4338, about 12.4 times the random baseline.
Task-Level Pattern
The task-layer heatmap shows that every task has at least one positive layer-level Spearman alignment, with the strongest task-specific values on list_summation and sorting_algorithms. The best layer by task is:
| Task | Best layer | Best Spearman alignment |
|---|---|---|
list_summation | 0 | 0.6070 |
sorting_algorithms | 3 | 0.3975 |
linear_equation_solving | 3 | 0.2701 |
base_conversion | 5 | 0.2430 |
grid_pathfinding | 7 | 0.2068 |
multidigit_addition | 7 | 0.1491 |
Artifacts
The generated trace and summary artifacts are in path:experiments/synthetic_sequences/disentangled/analysis/aggregates/continuous_cvae-Apr24-reconstruction-stack_20260424/data/attention_pattern_traces.jsonl, path:experiments/synthetic_sequences/disentangled/analysis/aggregates/continuous_cvae-Apr24-reconstruction-stack_20260424/data/attention_pattern_token_layer.csv, and the corresponding attention_pattern_*_summary.csv files. The replay script is path:experiments/synthetic_sequences/disentangled/analysis/run_continuous_cvae_apr22_attention_pattern.py, run with Apr24 report and loss-setting arguments.
