Qwen25 Multi-Task Paper-Method Grid: Results

Note

This file is observational. Interpretive claims and recommendations belong in analysis. The completed follow-up snapshot uses final-test metrics where metrics.json is present.

Collection State

The integrated snapshot has 39 expected cells across six collections: 39 have final metrics and 0 remain without final metrics.

collection_label	model_label	expected_cells	final_metric_cells	eval_snapshot_cells	COMPLETED
0.5B paper grid z64	Qwen2.5 0.5B	8	8	8	8
1.5B paper grid z64	Qwen2.5 1.5B	8	8	8	8
0.5B tokcc/dist sweep z8	Qwen2.5 0.5B	6	6	6	6
0.5B tokcc/dist sweep z64	Qwen2.5 0.5B	3	3	3	3
ratio/tokinv follow-up z64	Qwen2.5 0.5B	2	2	2	2
ratio/tokinv follow-up z64	Qwen2.5 1.5B	4	4	4	4
raw beta baseline z64	Qwen2.5 0.5B	4	4	4	4
raw beta baseline z64	Qwen2.5 1.5B	4	4	4	4

Completed Qwen2.5 0.5B Rows

Rows are ranked by matched router score.

collection_label	continuous_latent_dim	loss_setting	matched_router_score	analogical_set_overlap	final_answer_accuracy	within_task_router_probe	all_task_router_probe	strategy_match_rate	posterior_probe	reconstruction_loss	kl_prior_loss
0.5B tokcc/dist sweep z64	64	dist_plus_tokcc_a0p25_w0p75_tokcc0p25_b0p01_lin100	0.8502	0.9072	0.995	0.8574	0.2242	0.9916	0.9913	0.0006	2.3351
0.5B tokcc/dist sweep z8	8	dist_plus_tokcc_a0p25_w0p75_tokcc0p25_b0p01_lin100	0.7967		0.9969	0.8014	0.2016	0.9941	0.8131	0.0018	2.1244
0.5B paper grid z64	64	tokcc_a0p25_b0p01_lin100	0.7952	0.8252	0.9938	0.8092	0.2704	0.9827	0.974	0.5019	5.0621
0.5B paper grid z64	64	dist_norm_b0p01_lin100	0.7575	0.7471	0.998	0.76	0.2028	0.9968	0.9642	0.0108	1.5638
ratio/tokinv follow-up z64	64	dist0p25_tokcc0p75_a0p25_b0p01_lin100	0.7388	0.6377	0.9967	0.7447	0.2035	0.9921	0.9062	0.0147	2.5455
0.5B tokcc/dist sweep z8	8	tokinv_a0p25_b0p01_lin100	0.7259		0.9964	0.7301	0.1963	0.9942	0.8483	0.5487	2.6247
0.5B tokcc/dist sweep z8	8	dist_plus_tokcc_a0p25_w0p05_tokcc0p95_b0p01_lin100	0.7182		0.9936	0.7295	0.1854	0.9844	0.6831	0.0008	1.7038
raw beta baseline z64	64	raw_b0p01_const	0.7175	0.3447	0.9992	0.7181	0.1659	0.9992	0.7839	0.0321	1.9304
0.5B tokcc/dist sweep z8	8	dist_norm_b0p01_lin100	0.7141		0.9965	0.7171	0.183	0.9958	0.6656	0.0027	1.7813
raw beta baseline z64	64	raw_b0p1_const	0.7026	0.3740	0.9989	0.7034	0.1678	0.9989	0.769	0.0323	5.836
0.5B tokcc/dist sweep z8	8	tokcc_a0p25_b0p01_lin100	0.695		0.9776	0.7177	0.1941	0.9684	0.6838	0.53	1.9834
0.5B tokcc/dist sweep z64	64	dist_plus_tokcc_a0p25_w0p5_tokcc0p5_b0p01_lin100	0.6949	0.6777	0.9983	0.6965	0.2004	0.9976	0.9351	0.0172	1.6825
raw beta baseline z64	64	raw_b1_const	0.684	0.3506	0.9971	0.6867	0.149	0.9961	0.776	0.0331	1.7719
ratio/tokinv follow-up z64	64	tokinv_a1_eps1em8_b0p01_lin100	0.6776	0.3516	0.9991	0.6783	0.1407	0.9989	0.6927	0.5819	78.6067
raw beta baseline z64	64	raw_b1_lin100	0.6708	0.3545	0.9984	0.672	0.1603	0.9982	0.7899	0.0327	3.342
0.5B tokcc/dist sweep z8	8	dist_plus_tokcc_a0p25_w0p5_tokcc0p5_b0p01_lin100	0.6467		0.9983	0.6518	0.1821	0.9921	0.6864	0.0075	1.4014
0.5B paper grid z64	64	tokcc_a0p25_b0p1_lin100	0.592	0.3447	0.9996	0.5924	0.1509	0.9994	0.7764	0.6479	2.6485
0.5B paper grid z64	64	dist_plus_tokcc_a0p25_w0p05_tokcc0p95_b0p01_lin100	0.577	0.3613	0.9991	0.578	0.148	0.9984	0.7896	0.0321	0.7372
0.5B paper grid z64	64	dist_norm_b0p1_lin100	0.573	0.3594	0.9995	0.5733	0.1261	0.9995	0.7207	0.032	0.0086
0.5B paper grid z64	64	raw_b0p01_lin100	0.5719	0.3408	0.9995	0.5723	0.1366	0.9994	0.7768	0.032	0.6675
0.5B paper grid z64	64	dist_plus_tokcc_a0p25_w0p05_tokcc0p95_b0p1_lin100	0.5686	0.3662	0.9994	0.5692	0.1208	0.999	0.7266	0.0321	0.0103
0.5B tokcc/dist sweep z64	64	tokinv_a0p25_b0p01_lin100	0.5685	0.3535	0.9995	0.5688	0.1251	0.9995	0.8551	0.5858	0.0191
0.5B paper grid z64	64	raw_b0p1_lin100	0.565	0.3486	0.9995	0.5653	0.1235	0.9995	0.6859	0.032	0.0062

Tokcc/Dist-Weight Follow-Up

The z8 and z64 follow-up rows share six b0p01 settings. The integrated z64 frontier also includes the completed 25:75 and token-inverse-surprisal additions.

loss_setting	z8_matched_router_score	z64_matched_router_score	delta_z64_minus_z8_matched_router_score	z8_within_task_router_probe	z64_within_task_router_probe	z8_posterior_probe	z64_posterior_probe
dist_norm_b0p01_lin100	0.7141	0.7575	0.0434	0.7171	0.76	0.6656	0.9642
tokcc_a0p25_b0p01_lin100	0.6950	0.7952	0.1003	0.7177	0.8092	0.6838	0.974
dist_plus_tokcc_a0p25_w0p05_tokcc0p95_b0p01_lin100	0.7182	0.577	-0.1411	0.7295	0.578	0.6831	0.7896
dist0p25_tokcc0p75_a0p25_b0p01_lin100		0.7388			0.7447		0.9062
dist_plus_tokcc_a0p25_w0p5_tokcc0p5_b0p01_lin100	0.6467	0.6949	0.0482	0.6518	0.6965	0.6864	0.9351
dist_plus_tokcc_a0p25_w0p75_tokcc0p25_b0p01_lin100	0.7967	0.8502	0.0535	0.8014	0.8574	0.8131	0.9913
tokinv_a0p25_b0p01_lin100	0.7259	0.5685	-0.1574	0.7301	0.5688	0.8483	0.8551
tokinv_a1_eps1em8_b0p01_lin100		0.6776			0.6783		0.6927

Probe Scope

The completed 0.5B rows distinguish the preferred equal-task average of within-task router probes from the global all-task router probe.

Qwen2.5 1.5B Completed Rows

The table below uses final metrics from the completed paper grid and the b0p01 ratio/tokinv follow-up.

collection_label	loss_setting	slurm_state	analysis_status	has_final_metrics	matched_router_score	analogical_set_overlap	final_answer_accuracy	within_task_router_probe	all_task_router_probe
1.5B paper grid z64	dist_plus_tokcc_a0p25_w0p05_tokcc0p95_b0p01_lin100	COMPLETED	complete_final	True	0.9167	0.9053	0.9941	0.9323	0.2113
1.5B paper grid z64	tokcc_a0p25_b0p1_lin100	COMPLETED	complete_final	True	0.8845	0.8174	0.9925	0.9084	0.2125
ratio/tokinv follow-up z64	dist_plus_tokcc_a0p25_w0p75_tokcc0p25_b0p01_lin100	COMPLETED	complete_final	True	0.8823	0.8506	0.9951	0.891	0.2128
1.5B paper grid z64	dist_norm_b0p01_lin100	COMPLETED	complete_final	True	0.8086	0.8691	0.9958	0.8144	0.2138
ratio/tokinv follow-up z64	dist_plus_tokcc_a0p25_w0p5_tokcc0p5_b0p01_lin100	COMPLETED	complete_final	True	0.8049	0.6973	0.9956	0.8101	0.2069
ratio/tokinv follow-up z64	dist0p25_tokcc0p75_a0p25_b0p01_lin100	COMPLETED	complete_final	True	0.6958	0.5498	0.9987	0.6975	0.1647
raw beta baseline z64	raw_b0p01_const	COMPLETED	complete_final	True	0.6907	0.3555	0.9993	0.6912	0.1341
1.5B paper grid z64	dist_plus_tokcc_a0p25_w0p05_tokcc0p95_b0p1_lin100	COMPLETED	complete_final	True	0.6872	0.3779	0.9991	0.6879	0.1349
raw beta baseline z64	raw_b1_const	COMPLETED	complete_final	True	0.6861	0.3574	0.9993	0.6866	0.1294
1.5B paper grid z64	tokcc_a0p25_b0p01_lin100	COMPLETED	complete_final	True	0.6851	0.3564	0.9985	0.6868	0.1367
raw beta baseline z64	raw_b1_lin100	COMPLETED	complete_final	True	0.6835	0.3652	0.9994	0.6839	0.136
raw beta baseline z64	raw_b0p1_const	COMPLETED	complete_final	True	0.6737	0.3662	0.9994	0.6741	0.1309
ratio/tokinv follow-up z64	tokinv_a1_eps1em8_b0p01_lin100	COMPLETED	complete_final	True	0.6284	0.3721	0.9937	0.6341	0.1357
1.5B paper grid z64	dist_norm_b0p1_lin100	COMPLETED	complete_final	True	0.5844	0.3711	0.9994	0.5847	0.1333
1.5B paper grid z64	raw_b0p1_lin100	COMPLETED	complete_final	True	0.5689	0.3594	0.9995	0.5692	0.1283
1.5B paper grid z64	raw_b0p01_lin100	COMPLETED	complete_final	True	0.5566	0.3682	0.9994	0.5569	0.1239

1.5B within-task versus global router probe

Analogical Steering

Analogical steering metrics are loaded from each training run's analogical_steering/summary.json artifact when present, with the post-hoc z64 analogical-steering collection used as a fallback for older evaluated rows. The primary analogical metric is analogical_steering/pair_set_overlap_rate.

model_label	expected_cells	cells_with_final_metrics	cells_with_analogical_metrics	analogical_jobs	completed_analogical_jobs
Qwen2.5 0.5B	23	23	17	17	17
Qwen2.5 1.5B	16	16	16	16	16

model_label	collection_label	loss_setting	analogical_set_overlap	analogical_set_overlap_given_both_nonempty	analogical_unique_agreement	analogical_both_unique	analogical_both_answer_correct	matched_router_score
Qwen2.5 0.5B	0.5B tokcc/dist sweep z64	dist_plus_tokcc_a0p25_w0p75_tokcc0p25_b0p01_lin100	0.9072	0.9198	0.9201	0.9043	0.9922	0.8502
Qwen2.5 1.5B	1.5B paper grid z64	dist_plus_tokcc_a0p25_w0p05_tokcc0p95_b0p01_lin100	0.9053	0.9279	0.9233	0.8789	0.9893	0.9167
Qwen2.5 1.5B	1.5B paper grid z64	dist_norm_b0p01_lin100	0.8691	0.8777	0.8721	0.8936	0.9941	0.8086
Qwen2.5 1.5B	ratio/tokinv follow-up z64	dist_plus_tokcc_a0p25_w0p75_tokcc0p25_b0p01_lin100	0.8506	0.8667	0.8648	0.8955	0.9893	0.8823
Qwen2.5 0.5B	0.5B paper grid z64	tokcc_a0p25_b0p01_lin100	0.8252	0.8587	0.8582	0.8955	0.9922	0.7952
Qwen2.5 1.5B	1.5B paper grid z64	tokcc_a0p25_b0p1_lin100	0.8174	0.8848	0.8789	0.8389	0.9727	0.8845
Qwen2.5 0.5B	0.5B paper grid z64	dist_norm_b0p01_lin100	0.7471	0.7507	0.7412	0.8906	0.9971	0.7575
Qwen2.5 1.5B	ratio/tokinv follow-up z64	dist_plus_tokcc_a0p25_w0p5_tokcc0p5_b0p01_lin100	0.6973	0.7055	0.6913	0.9111	0.9922	0.8049
Qwen2.5 0.5B	0.5B tokcc/dist sweep z64	dist_plus_tokcc_a0p25_w0p5_tokcc0p5_b0p01_lin100	0.6777	0.6784	0.6652	0.8984	0.999	0.6949
Qwen2.5 0.5B	ratio/tokinv follow-up z64	dist0p25_tokcc0p75_a0p25_b0p01_lin100	0.6377	0.6498	0.6186	0.8604	0.9941	0.7388
Qwen2.5 1.5B	ratio/tokinv follow-up z64	dist0p25_tokcc0p75_a0p25_b0p01_lin100	0.5498	0.5525	0.4989	0.8613	0.998	0.6958
Qwen2.5 1.5B	1.5B paper grid z64	dist_plus_tokcc_a0p25_w0p05_tokcc0p95_b0p1_lin100	0.3779	0.3798	0.3517	0.8857	0.9961	0.6872

The correlation table below compares analogical set-overlap with the core z64 final-test metrics.

model_label	comparison_metric_label	n_runs	pearson_r
Qwen2.5 0.5B	matched router score	17	0.8104
Qwen2.5 0.5B	within-task router probe	17	0.818
Qwen2.5 0.5B	all-task router probe	17	0.8977
Qwen2.5 0.5B	strategy-match rate	17	-0.7724
Qwen2.5 0.5B	final answer accuracy	17	-0.8276
Qwen2.5 0.5B	reconstruction loss	17	-0.094
Qwen2.5 0.5B	distribution-normalized reconstruction	17	-0.0908
Qwen2.5 0.5B	TOK-CC reconstruction	17	-0.9418
Qwen2.5 0.5B	token-inverse reconstruction	9	-0.4363
Qwen2.5 1.5B	matched router score	16	0.9027
Qwen2.5 1.5B	within-task router probe	16	0.9058
Qwen2.5 1.5B	all-task router probe	16	0.984
Qwen2.5 1.5B	strategy-match rate	16	-0.7615
Qwen2.5 1.5B	final answer accuracy	16	-0.768
Qwen2.5 1.5B	reconstruction loss	16	-0.0492
Qwen2.5 1.5B	distribution-normalized reconstruction	16	-0.0453
Qwen2.5 1.5B	TOK-CC reconstruction	16	-0.9321
Qwen2.5 1.5B	token-inverse reconstruction	8	-0.5989
All z64 analogical rows	matched router score	33	0.8583
All z64 analogical rows	within-task router probe	33	0.863
All z64 analogical rows	all-task router probe	33	0.9143
All z64 analogical rows	strategy-match rate	33	-0.7555
All z64 analogical rows	final answer accuracy	33	-0.787
All z64 analogical rows	reconstruction loss	33	-0.0762
All z64 analogical rows	distribution-normalized reconstruction	33	-0.0728
All z64 analogical rows	TOK-CC reconstruction	33	-0.9372
All z64 analogical rows	token-inverse reconstruction	17	-0.3503

Analogical Task Means

model_label	task_name	pair_unique_agreement_rate
Qwen2.5 0.5B	base_conversion	0.5163
Qwen2.5 0.5B	grid_pathfinding	0.5355
Qwen2.5 0.5B	linear_equation_solving	0.4447
Qwen2.5 0.5B	list_summation	0.4322
Qwen2.5 0.5B	multidigit_addition	0.5348
Qwen2.5 0.5B	sorting_algorithms	0.2627
Qwen2.5 1.5B	base_conversion	0.5283
Qwen2.5 1.5B	grid_pathfinding	0.5834
Qwen2.5 1.5B	linear_equation_solving	0.5575
Qwen2.5 1.5B	list_summation	0.4839
Qwen2.5 1.5B	multidigit_addition	0.5499
Qwen2.5 1.5B	sorting_algorithms	0.3383

Repo-Side Artifacts

Aggregate output directory: path:experiments/synthetic_sequences/disentangled/analysis/aggregates/synthetic-pretrained-disentangled-from-entangled-v2-paper-method-grid-multi-task-z64-qwen25_20260501
Report: path:experiments/synthetic_sequences/disentangled/analysis/aggregates/synthetic-pretrained-disentangled-from-entangled-v2-paper-method-grid-multi-task-z64-qwen25_20260501/report.md
Findings JSON: path:experiments/synthetic_sequences/disentangled/analysis/aggregates/synthetic-pretrained-disentangled-from-entangled-v2-paper-method-grid-multi-task-z64-qwen25_20260501/findings.json
Notebook: path:experiments/synthetic_sequences/disentangled/analysis/aggregates/synthetic-pretrained-disentangled-from-entangled-v2-paper-method-grid-multi-task-z64-qwen25_20260501/notebook.ipynb