Representation-Based Exploration for Language Models: From Test-Time to Post-Training

Citation

Authors: Jens Tuyls, Dylan J. Foster, Akshay Krishnamurthy, and Jordan T. Ash
Year: 2025
Venue: arXiv preprint
arXiv: arXiv:2510.11686
URL: https://arxiv.org/abs/2510.11686

One-Line Takeaway

RepExp uses hidden-state geometry to reward novelty, with the goal of moving RL for language models from mere sharpening toward deliberate discovery of diverse successful behaviors.

Technical Core

For a prompt-response pair (x, y), the method builds a representation h_theta(x, y) from averaged last-layer hidden states, then uses a random projection plus an elliptical bonus to score novelty relative to previously selected responses.
The same representation-based bonus is used in two places: inference-time subset selection from a candidate pool, and RL post-training by augmenting the extrinsic verifiable reward.
The intended effect is to preserve or widen solution diversity rather than collapsing onto a narrow family of rewarded traces.

Method

In the test-time setting, sample many candidates from the base model, then greedily select a smaller diverse subset using a history-aware novelty bonus.
In RL training, replace a pure reward r*(x, y) with an augmented reward r*(x, y) + beta b(x, y) where b is the batchwise representation bonus.
The method relies on the base model's learned internal features to define novelty rather than on token-level entropy alone.

Empirical Evaluation

RepExp improves verifier efficiency in best-of-N style inference, especially for stronger models and harder questions.
In RL post-training, it mitigates the diversity-collapse pattern seen in standard GRPO and preserves higher-k pass rates more effectively than sharpening-oriented baselines.

Assumptions And Limits

The method assumes the model's hidden-state geometry is informative enough for novelty search. The reported gains are strongest when the base model already has reasonably good internal representations; weak models may not benefit.

Relevance To This Project

This is an especially relevant exploration baseline because it attacks the same bottleneck as the repo: the failure of passive RLVR to reach new strategies. The difference is that RepExp works directly in representation space, whereas this project is trying to recover a disentangled latent strategy variable that would make exploration and intervention more explicit.

Comparison Notes

Yue et al. (2025) and Wu et al. (2025) diagnose support-constrained sharpening; this paper is an algorithmic attempt to escape that regime. Compared with POPE, RepExp uses unsupervised novelty signals from the model's own representations rather than privileged oracle prefixes. Compared with BroRL, it broadens exploration by diversity-aware selection instead of just increasing rollout count.

Source Note

Retrieval: alphaXiv MCP get_paper_content on arXiv:2510.11686; no local cache or downloaded source artifacts were created.

Verified from source: verification in this ingestion pass was limited to the alphaXiv whole-paper report and arXiv metadata exposed through that report. I did not run a separate targeted PDF query for this note.