OLMo 2 and Gemma 3 Compatibility Validation

Back to design

Environment

Validation used PyTorch 2.10.0+cu128, Transformers 5.2.0, and PEFT 0.18.1. CUDA was unavailable: torch.cuda.is_available() returned false and nvidia-smi could not communicate with the NVIDIA driver. The completed checks are therefore CPU checks, not CUDA validation.

OLMo 2 1B

Checkpoint: allenai/OLMo-2-0425-1B.

The official tokenizer and complete two-shard checkpoint were downloaded. The tokenizer exposes pad token ID 100277 and EOS token ID 100257. The official configuration reports model type olmo2, hidden size 2048, 16 layers, and checkpoint vocabulary size 100352.

All requested standard LoRA targets occur in the actual checkpoint:

The tokenizer has 100,278 entries before adding the five synthetic control tokens, while the checkpoint reserves 100,352 embedding rows. Adding the control tokens produces tokenizer size 100,283, with <z> at 100280 and <z_empty> at 100281. Both IDs remain within the checkpoint's existing embedding range. The current runtime's resize call will shrink the reserved vocabulary tail to the tokenizer size rather than grow the embedding matrix; the retained tokenizer IDs, pad token, EOS token, and new control-token IDs all remain in range.

A reduced-dimension Olmo2ForCausalLM using the real tokenizer completed:

• embedding and output-head interface checks;
• LoRA attachment to all seven targets with sparse control-token adaptation;
• a z=64 continuous router forward pass;
• latent injection through inputs_embeds;
• reconstruction-loss backward;
• non-null gradients for LoRA parameters and z_to_hidden.

The full 1B checkpoint could not be materialized in this devbox execution environment: loading was killed with exit 137, including when requested in bfloat16. Actual-weight forward/backward and CUDA execution remain unvalidated locally.

Gemma 3 270M

Checkpoint: google/gemma-3-270m.

Gated-repository access was confirmed after authenticating the devbox with the approved Hugging Face account. The official tokenizer and checkpoint were downloaded and loaded locally. The configuration reports model type gemma3_text, hidden size 640, intermediate size 2048, 18 layers, four attention heads, one key/value head, and checkpoint vocabulary size 262144.

The tokenizer exposes pad token ID 0 and EOS token ID 1. It has 262,145 entries before adding the five synthetic control tokens, one more than the checkpoint embedding vocabulary. Adding the control tokens and applying the runtime resize produces 262,150 tied input/output embedding rows, with <z> at 262147 and <z_empty> at 262148.

The real float32 Gemma3ForCausalLM completed:

• tokenizer expansion and tied input/output embedding resize;
• LoRA attachment to q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, and down_proj;
• a z=64 continuous router forward pass;
• latent injection through inputs_embeds;
• reconstruction-loss backward;
• non-null gradients for LoRA parameters and z_to_hidden.

The first CPU attempt loaded the backbone in bfloat16 and failed when its hidden states reached newly initialized float32 CVAE heads without autocast. Reloading the model in float32 completed successfully. The production H100 path requests bfloat16 CUDA autocast, but that path remains unvalidated on this GPU-less devbox.

Authentication is resolved on the devbox. Before submission, verify that the saved Hugging Face credential is visible from the Slurm compute environment without copying the token into a checked-in spec or job script.

Gemma 3 1B PT

Checkpoint: google/gemma-3-1b-pt.

Gated access was confirmed with the saved Hugging Face credential, and the complete 2.0 GB checkpoint plus tokenizer were downloaded to the shared cache. The official configuration reports model type gemma3_text, hidden size 1152, intermediate size 6912, 26 layers, four attention heads, one key/value head, and checkpoint vocabulary size 262144. All seven standard LoRA targets occur in the resolved Gemma3ForCausalLM architecture.

The tokenizer exposes pad token ID 0 and EOS token ID 1. It has 262,145 entries before augmentation. Adding the six structural tokens and <z_empty> produces 262,152 tokenizer entries and tied embedding rows, with <z> at 262149 and <z_empty> at 262151.

The full bfloat16 checkpoint materialized successfully and reached embedding resizing plus LoRA setup. A complete 26-layer CPU forward/backward remained compute-bound and was stopped after approximately three minutes without an exception or memory failure. The approved fallback retained the real tokenizer, vocabulary, hidden size, intermediate size, module layout, and all seven LoRA targets while reducing the transformer to one layer. It completed:

• tied embedding/output resizing to 262,152 rows;
• sparse control-token adaptation and LoRA attachment;
• z=64 latent injection through inputs_embeds;
• reconstruction-loss backward;
• non-null gradients for LoRA parameters and z_to_hidden.

The fallback used 850 MB peak resident memory and completed in 2 minutes 20 seconds. CUDA remains unavailable on the devbox, so production bfloat16 CUDA execution is not locally validated.

Status