synthetic-pretrained-entangled-v1 Design

Motivation

synthetic-pretrained-entangled-v1 establishes the pretrained-backed entangled source model for the synthetic-sequences line. The design goal is not merely to solve the tasks, but to produce a strong mixed-strategy model whose outputs and hidden states can later be used as the source distribution for disentangling experiments.

This collection is the entangled-side counterpart to the later disentangling comparison notes in ss-disentangled-direct-pretrained-v1 and synthetic-pretrained-disentangled-from-entangled-v1.

Experiment Metadata

• Collection: synthetic-pretrained-entangled-v1
• Slurmkit collection: path:.slurmkit/collections/ss-entangled-pretrained-v1.yaml
• Source run directory: path:.runs/synthetic_sequences/entangled/synthetic-pretrained-entangled-v1
• Collection commit ID: completed runs report git commit c2de547
• Planned design: 6 tasks x 1 pretrained model x 1 seed = 6 jobs
• Observed completion at analysis time: 6/6 task runs analyzable, with 2 abandoned retry directories on base_conversion

Training Design

• Backbone family: pretrained Qwen2.5-0.5B
• Adaptation mode: lora
• LoRA configuration: rank 16, alpha 32, dropout 0.0
• Model size preset: s (hidden_size=384, num_hidden_layers=6)
• Training budget: 1 epoch, batch_size=128, eval_batch_size=128
• Optimization: adamw, lr=2e-4, linear-warmup cosine decay, weight_decay=0.0
• Evaluation cadence: val_check_interval=0.02, sampled generation always on, greedy generation off
• Probe configuration: enabled, probe_train_size=256, probe_eval_size=256, probe_max_relative_position=64

Sweep Axes

This is a fixed-config, cross-task collection rather than a hyperparameter sweep.

• Swept axis: task_name
• Tasks:
  • list_summation
  • sorting_algorithms
  • grid_pathfinding
  • linear_equation_solving
  • base_conversion
  • multidigit_addition

Research Questions

1. Does the pretrained entangled source model preserve a real mixture over strategies while maintaining near-saturated task accuracy?
2. How task-dependent are strategy uniqueness, ambiguity, and hidden-state strategy decodability in the final source model?
3. How do answer accuracy, mixture quality, and probe decodability evolve during training, and which tasks look strongest as future source checkpoints?

Primary Metrics

• sampled test final-answer accuracy
• sampled test unique-strategy rate
• sampled test ambiguous-strategy rate
• final mean probe accuracy
• best validation loss

Secondary Metrics

• sampled test parse-success rate
• sampled test off-strategy rate
• sampled test invalid-structure rate
• sampled validation final-answer accuracy
• sampled validation unique-strategy rate

Success Criteria

• all planned tasks finish with analyzable artifacts
• sampled final-answer accuracy remains near-saturated across tasks
• the collection preserves nontrivial multi-strategy behavior rather than collapsing uniformly to one strategy
• probe diagnostics remain strong enough to support later source-model comparisons