Additional Model Families Design
Additional Model Families Design
Scope
This experiment series tests whether the paper-selected multi-task factorizations transfer from Qwen2.5 to other pretrained model families at similar scale. The first models were Pythia-410M and SmolLM2-360M. Follow-up models are OLMo 2 1B, Gemma 3 270M, and the Gemma 3 1B pretrained checkpoint.
Shared settings:
- seed
314159; - one epoch, up to
200000sequences; - six paper roles: Standard CVAE, β-ELBO, Global normalization, Token weighting, Global + token, and Token inverse;
- analogical transfer evaluated once after factorization training.
The Pythia LoRA targets were query_key_value, dense, dense_h_to_4h, and dense_4h_to_h; SmolLM2 used the standard attention and MLP projection targets from its registry entry.
Each model is represented by an entangled SFT execution collection and a dependent z=64 factorization collection containing the six paper-selected roles. These execution collections remain separate so the scheduler can express dependencies and parallelize the factorization jobs; their results are documented together in this report.
Source-Learning Prerequisite
Factorization results are eligible for method comparison only when the entangled source learns the benchmark behavior. This report uses test strategy match >=0.95 as the operational prerequisite. SmolLM2 passes at 0.9937; Pythia fails at one epoch (0.6051) and remains below the threshold after the three-epoch follow-up (0.8376). Pythia is therefore retained only as a diagnostic in pythia.
Follow-Up Models
OLMo 2 1B, Gemma 3 270M, and Gemma 3 1B PT use the same paper-role settings after local architecture, tokenizer, LoRA, and factorization-interface validation. All three use the standard q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, and down_proj LoRA targets. Their source-learning results must pass the same prerequisite before their factorizations are treated as evidence about the method.
