Synthetic Sequences Evaluation And Diagnostics

The synthetic-sequences setting is designed so disentanglement can be evaluated at the level of answer quality, strategy attribution, and latent controllability. The core evaluation question is whether the learned latent variable carries reusable strategy information rather than only preserving task accuracy.

Main Evaluation Modes

The experiment package uses three main evaluation surfaces:

• router-sampled generation, which asks what the model does when it chooses its own latent,
• controlled-latent generation, which asks what behavior each latent induces when forced,
• posterior diagnostics, which ask whether strategy can be recovered from the posterior latent representation.

A planned standalone exploration-efficiency evaluation adds a fourth surface for continuous checkpoints: for each sampled test input, draw a sequence of latent states from the router and measure how quickly parsed generated responses cover the task's strategy set. The first version should compare iid router sampling against kernel-reweighted router sampling, using the loading and generation pattern from path:experiments/synthetic_sequences/disentangled/analogical_steering.py.

The implemented evaluation contract is documented in path:experiments/synthetic_sequences/disentangled/docs/supervision_and_eval.md.

Core Metric Families

The most important metric groups are:

• answer quality: parse success and final-answer accuracy,
• discrete latent structure: one-to-one alignment and local strategy coverage,
• continuous latent structure: sampled-latent probe accuracy, centroid separation, and posterior-mu probe accuracy,
• optimization diagnostics: reconstruction, normalized reconstruction, KL, and auxiliary-loss contributions.

These metrics intentionally separate "solves the task" from "uses the latent in a strategy-meaningful way."

Discrete Vs Continuous Diagnostic Split

The discrete and continuous settings currently expose slightly different evaluation strengths:

• discrete mode supports controlled-latent generation directly,
• continuous mode currently emphasizes router-sampled generation and posterior-moment diagnostics,
• continuous mode also emits latent-geometry diagnostics and animated validation artifacts that do not have a direct discrete counterpart.

This means comparisons across latent types should focus on shared metrics first, then treat type-specific diagnostics as supplementary evidence.

Why The Diagnostics Are High Yield

Because the tasks have exact validators and known strategy taxonomies, failures can be localized precisely:

• the model may answer correctly while collapsing strategies,
• the posterior may preserve strategy information while the router does not,
• or the latent may control behavior locally without supporting a single global mapping.

That diagnostic separation is the main experimental advantage of the synthetic setting.

Planned Exploration-Efficiency Evaluation

The first exploration-efficiency script should live under path:experiments/synthetic_sequences/disentangled/ and evaluate continuous disentangled checkpoints only. It should sample num_examples records randomly without replacement from the requested split, defaulting to test, then run num_samples_per_input latent samples for each method and input.

The methods are:

• iid: iid samples from the continuous router;
• kernel_reweighted: kernel-reweighted rejection sampling in whitened router coordinates with SIR fallback.

Strategy coverage should be computed from ValidationResult.strategy_matches returned by task parsing of each generated response. Coverage is set-valued: if one response matches multiple strategies, all matched strategies count as covered. The denominator is len(task.strategies()).

Artifacts should be written under run_dir/exploration_efficiency/sig-{kernel_sigma}_beta-{kernel_beta}_eps-{acceptance_floor}/. The full resolved configuration should be stored in config.json, with raw sequence records in records.jsonl, aggregate metrics in summary.json, and plots for coverage, novelty, full coverage, and kernel diagnostics. Multi-task runs should include root metrics across all sampled examples plus by_task/<task_name>/summary.json and corresponding plots.