Apr22 Beta-Schedule Profiling
Apr22 Beta-Schedule Profiling
Back to design Back to results
This note is scoped to local profiling artifacts from path:.runs/synthetic_sequences/disentangled/exp2_continuous_cvae-Apr22-beta-schedule-followup/*/profiling/profiling_summary.json and aggregate runtime summaries under path:experiments/synthetic_sequences/disentangled/analysis/aggregates/continuous_cvae-Apr22-beta-schedule-followup_20260422. Runtime here means profiled in-process wall time from the run profiler, not scheduler-reported elapsed time. All 108/108 Apr22 runs have profiling summaries.
Runtime Snapshot
- profiled runs:
108 - mean profiled runtime:
3.25h - median profiled runtime:
2.85h - range:
0.56hto8.02h - slowest mean tasks:
base_conversion6.20h,sorting_algorithms5.89h,grid_pathfinding3.10h - fastest mean tasks:
list_summation0.63h,linear_equation_solving1.13h,multidigit_addition2.54h - mean runtime by adaptation mode:
lora3.80h,full3.04h,reinit2.91h
Caption: Mean runtime in hours by loss setting and adaptation mode.
Runtime By Loss Setting
Runtime differences are smaller than the main metric differences, but the secondary runtime table is still structured rather than flat:
beta0p01_d50_lin50is the shortest mean-runtime setting overall at2.9081h.beta0p1_cyclin25_w12p5is the longest mean-runtime setting overall at3.4135h.- The largest single matched runtime difference is the
beta=0.01 / loracomparison, where delayed linear runs about1.1352hfaster than the matched cyclical schedule.
| loss_setting | mean_runtime_hours |
|---|---|
| beta0p1_cyclin25_w12p5 | 3.4135 |
| beta0p1_d50_lin50 | 3.3307 |
| beta1_d50_lin50 | 3.3190 |
| beta0p01_cyclin25_w12p5 | 3.3043 |
| beta0_unnorm | 3.2233 |
| beta0p01_d50_lin50 | 2.9081 |
Where Time Goes
The run profiler is dominated by scheduled router-sampled evaluation. Averaged over runs, epoch/eval_router_sampled accounts for 43.04% of profiler total, and epoch/eval_router_sampled/free_generation alone accounts for 41.74%. The next largest sections are much smaller: run/setup 4.89%, epoch/train 4.02%, epoch/eval_val_loss 3.05%, and scheduled posterior_mu_diag 1.24%.
| profiler_section | mean_total_hours | mean_share_of_profile_total | mean_count | mean_event_seconds |
|---|---|---|---|---|
| epoch/eval_router_sampled | 1.4672 | 43.04% | 50.3 | 106.40 |
| epoch/eval_router_sampled/free_generation | 1.4361 | 41.74% | 50.3 | 104.16 |
| run/setup | 0.0946 | 4.89% | 1.0 | 340.50 |
| epoch/train | 0.0875 | 4.02% | 50.3 | 6.28 |
| epoch/eval_val_loss | 0.0702 | 3.05% | 50.3 | 5.06 |
| epoch/eval_router_sampled/posterior_mu_diag | 0.0300 | 1.24% | 50.3 | 2.16 |
| run/final_eval_router_sampled | 0.0296 | 0.89% | 1.0 | 106.48 |
| run/final_eval_val_loss | 0.0020 | 0.09% | 1.0 | 7.17 |
| epoch/eval_router_sampled/router_sampled_z_animation | 0.0009 | 0.05% | 50.3 | 0.07 |
| epoch/checkpoint | 0.0001 | 0.01% | 1.0 | 0.32 |
The practical interpretation is that this collection is evaluation-bound rather than training-bound. The scheduled router-sampled generation pass repeats about 50 times per run and is much larger than the train window, validation-loss pass, checkpoint, output-writing, and animation costs.
Task Dependence
Task explains the clearest runtime spread. The slow tasks are slow because scheduled router-sampled generation is slow, not because training or validation loss changes enough to dominate.
| task_name | mean_runtime_hours | mean_router_eval_seconds | mean_free_generation_seconds | mean_val_loss_seconds | mean_train_window_seconds |
|---|---|---|---|---|---|
| base_conversion | 6.20 | 217.7 | 214.5 | 7.0 | 8.3 |
| sorting_algorithms | 5.89 | 191.8 | 188.5 | 7.1 | 8.3 |
| grid_pathfinding | 3.10 | 101.1 | 98.7 | 5.3 | 6.7 |
| multidigit_addition | 2.54 | 78.9 | 76.8 | 4.9 | 6.1 |
| linear_equation_solving | 1.13 | 32.8 | 31.4 | 3.4 | 4.5 |
| list_summation | 0.63 | 16.2 | 15.1 | 2.6 | 3.7 |
Speedup Implications
The highest-leverage speedup knob is scheduled evaluation cost. Loosening val_check_interval would directly reduce the repeated epoch/eval_router_sampled and epoch/eval_val_loss sections; reducing the scheduled validation generation budget would target the single largest subsection, epoch/eval_router_sampled/free_generation.
Checkpointing, output writing, and router-sampled-z animation are low-leverage for this collection. Combined, they are far below 1% of profiler total, so optimizing those paths would not materially change sweep runtime.
