from-entangled-v2 continuous-v2 Profiling

Back to design Back to results

Runtime Snapshot

• profiled runs: 36
• mean profiled runtime: 10.26h
• median profiled runtime: 7.47h
• range: 3.10h to 36.11h
• shared runtime settings in this collection: val_check_interval=0.02, eval_generation_num_samples=None, batch_size=128, eval_batch_size=128, micro_batch_size=auto
• slowest mean tasks: base_conversion 19.05h, sorting_algorithms 17.19h, multidigit_addition 9.31h
• fastest mean tasks: list_summation 3.72h, linear_equation_solving 4.27h

Caption: Left: main-effect strength on profiled runtime hours. Right: mean profiled runtime by task and loss setting.

Caption: Per-task runtime distribution across the six loss settings, with overall mean and median reference lines.

What Runtime Depends On

Task drives most runtime variation in the current collection. The runtime-factor figure gives η²≈0.64 for task_name, compared with η²≈0.12 for loss_setting; the remaining single fields are much smaller (beta≈0.10, beta_schedule≈0.00, token_weighted_reconstruction_weight≈0.00). The task-by-loss heatmap also shows that the slowest cells cluster in base_conversion and sorting_algorithms, while beta0_const_pv0 is the slowest loss family on average at 15.01h.

At the per-eval level, the main difference between fast and slow tasks is the cost of router-sampled validation generation, not checkpointing or train-window compute. Mean router-sampled eval time is 618s for base_conversion and 582s for sorting_algorithms, versus 112s for list_summation and 128s for linear_equation_solving. The current validation cadence is also high in absolute terms: val_check_interval=0.02 yields 32 or 51 scheduled evals per run.

Caption: Mean per-eval train-window, val-loss, router-sampled eval, and checkpoint timing by task.

Where Time Goes In The Code Path

The profiler is dominated by scheduled router-sampled evaluation, not training. Averaged over runs, epoch/eval_router_sampled takes 44.06% of total runtime, and epoch/eval_router_sampled/free_generation alone accounts for 43.06%. The next-largest sections are much smaller: epoch/train 4.30%, epoch/eval_val_loss 3.45%, run/setup 1.54%, and run/final_eval_router_sampled 1.18%. Checkpointing and end-of-run writes are effectively negligible in the current profile.

Within scheduled router eval, free generation dominates every task. Mean scheduled free-generation time per eval ranges from 108s on list_summation to 609s on base_conversion, while posterior-mu diagnostics stay in the single-digit seconds and router-sampled-z animation stays near 0.34s per eval.

Caption: Runtime composition by major profiler section for each task and overall. Scheduled router evaluation dominates every task, with smaller contributions from training, validation-loss evaluation, setup, and final evaluation.

Caption: Mean scheduled router-eval subsection timing, highlighting the dominance of free generation over the smaller diagnostics.

Speedup Opportunities

The current measurements point to two high-leverage knobs and several low-leverage ones.

1. Increase val_check_interval if the goal is to reduce wall time with minimal code changes. The repeated scheduled-eval sections account for 47.51% of runtime, and the rough upper bound for moving toward epoch-end-only eval is 46.57% saved runtime because the current runs perform 32 or 51 scheduled evals each.
2. Reduce eval_generation_num_samples for scheduled validation generation. The current config leaves it unset, so scheduled validation free generation runs on the full selected subset. That section alone accounts for a rough upper bound of 43.06% runtime. This knob does not shrink final test evaluation, so it is a cleaner validation-time tradeoff than changing the final report path.
3. Deprioritize checkpoint and output tuning. Checkpointing plus output writes account for only 0.01% of runtime in the current profile.
4. Treat posterior-mu diagnostics and router-sampled-z animation as secondary levers. Their rough upper bounds are 0.96% and 0.07% respectively, so they are measurable but far below the two main validation knobs.

These percentages are rough upper bounds derived from the current profiler shares, not fresh rerun measurements. The main practical implication is still clear: this collection is evaluation-bound rather than training-bound. If future sweeps need to run faster, the first changes to test should be a looser validation cadence and a smaller scheduled-validation generation budget, not checkpoint or output-path micro-optimizations.