Exploration Efficiency
Exploration Efficiency
This note uses corrected task-local strategy coverage recomputed from records.jsonl. The original eval artifacts normalized coverage by the full multi-task strategy set, which is useful as a bug diagnostic but not as the paper-facing exploration-efficiency metric.
Artifact Links
- Slurmkit collection:
path:.slurmkit/collections/synthetic-pretrained/exploration-efficiency/from-entangled-v2/z64-qwen25-global-token-kernel-sweep.yaml - Aggregate output directory:
path:experiments/synthetic_sequences/disentangled/analysis/aggregates/synthetic-pretrained-exploration-efficiency-from-entangled-v2-z64-qwen25-global-token-kernel-sweep_20260604 - Repo-side report:
path:experiments/synthetic_sequences/disentangled/analysis/aggregates/synthetic-pretrained-exploration-efficiency-from-entangled-v2-z64-qwen25-global-token-kernel-sweep_20260604/report.md - Brief:
path:experiments/synthetic_sequences/disentangled/analysis/aggregates/synthetic-pretrained-exploration-efficiency-from-entangled-v2-z64-qwen25-global-token-kernel-sweep_20260604/brief.yaml - Findings JSON:
path:experiments/synthetic_sequences/disentangled/analysis/aggregates/synthetic-pretrained-exploration-efficiency-from-entangled-v2-z64-qwen25-global-token-kernel-sweep_20260604/findings.json - Notebook:
path:experiments/synthetic_sequences/disentangled/analysis/aggregates/synthetic-pretrained-exploration-efficiency-from-entangled-v2-z64-qwen25-global-token-kernel-sweep_20260604/notebook.ipynb
Collection State
The collection has 18 completed evaluation jobs: two selected source rows crossed with three kernel_sigma values and three kernel_beta values. The source rows are the Qwen2.5 0.5B global+token row from the z64 TOK-CC/dist sweep and the Qwen2.5 1.5B global+token row from the paper grid, so this is a selected-row comparison rather than a pure model-size ablation.
Corrected Coverage Result
The primary metric is task-local strategy coverage over 32 samples. For each generated sample sequence, the denominator is the number of strategies for the example's effective subtask.
| source row | mean AUC delta | best AUC delta | mean final-coverage delta | fallback rate | mean rejection attempts |
|---|---|---|---|---|---|
| Qwen2.5 0.5B selected row | -0.0030 | -0.0029 | -0.0003 | 0.0000 | 1.00 |
| Qwen2.5 1.5B selected row | -0.0064 | -0.0059 | -0.0010 | 0.0000 | 1.00 |
Task-Level View
The strongest task-level AUC delta was +0.0116 for multidigit_addition under Qwen2.5 0.5B selected row with sigma=0.75 and beta=5.
Guardrails
The kernel sampler's fallback rate was 0.0000 and its mean rejection attempts were 1.00 across the completed sweep.
Interpretation
On the corrected task-local metric, this completed kernel sweep is a null result for exploration gains: the best run has AUC delta -0.0029 and the worst has -0.0066. The guardrails suggest a concrete mechanism for the null result: the bounded rejection sampler accepted almost every first proposal, so the intended kernel repulsion had little opportunity to alter the router sample sequence.
The evaluation still supports the broader analysis workflow: task-local strategy coverage can be measured from external strategy labels, and the raw records are sufficient to repair metric-definition bugs without rerunning model generation. A follow-up should first fix or stress-test the kernel acceptance schedule before using this sampler as evidence about the limits of latent-space exploration.
Calibration Follow-Up
A CPU-only calibration pass over the saved whitened u samples indicates that the explored kernel grid was effectively inert in 64D. For a 32-sample budget, the largest swept value, sigma=1.0, has expected final-step RBF history density about 1.7e-14; the corresponding hard-ball affected mass is about 5e-54, so changing beta cannot materially alter acceptance. Dimension-aware calibration suggests follow-up runs should use substantially larger whitened radii, roughly sigma=3.2 to 4.2 for target final-step RBF densities between 0.1 and 1.0, with beta chosen from a target acceptance rate rather than fixed globally. The calibration artifacts are in path:experiments/synthetic_sequences/disentangled/analysis/aggregates/synthetic-pretrained-exploration-efficiency-from-entangled-v2-z64-qwen25-global-token-kernel-calibration_20260604, and follow-up evaluation runs should test this calibrated grid before treating the current null result as evidence against latent-space exploration itself.
