Exploration Efficiency Methods

This note records the narrowed set of adaptive latent-state sampling methods for using a router-generator factorization to explore strategy space more efficiently than iid base-model sampling. The corresponding evaluation metrics are in Exploration efficiency coverage metrics.

Related notes:

• Exploration efficiency coverage metrics
• Exploration efficiency kernel calibration
• Exploration efficiency gains
• Iterative self-improvement via analogical exploration

Continuous Gaussian Setup

The main case of interest is a continuous router

$$r(z|x) = \mathcal N(\mu_x, \Sigma_x).$$

All adaptive sampling should be formulated in whitened router coordinates. Let $L_x L_x^\top = \Sigma_x$ and write

$$z = \mu_x + L_x u, \qquad u \sim \mathcal N(0,I).$$

The method samples or selects $u_t$ in whitened coordinates, then maps back to $z_t = \mu_x + L_x u_t$ before decoding $y_t \sim g(\cdot|x,z_t)$. This avoids confusing exploration with the router mean, coordinate scaling, or covariance anisotropy.

The history is

$$H_{t-1} = \{u_1,\ldots,u_{t-1}\}.$$

Latent-space exploration is only a proposal mechanism. Evaluation should still use external strategy labels as described in Exploration efficiency coverage metrics.

Method 1: iid Router Sampling (baseline)

The baseline samples directly from the router:

$$u_t \sim \mathcal N(0,I), \qquad z_t = \mu_x + L_x u_t.$$

This is the reference policy for measuring whether adaptive latent-state sampling improves strategy coverage over ordinary iid base-model sampling.

Method 2: Inverse Elliptical Candidate Reranking

At step $t$, draw a candidate pool from the router:

$$u^{(1)},\ldots,u^{(K)} \sim \mathcal N(0,I).$$

Define the history design matrix

$$V_t = \lambda I + \sum_{i<t} u_i u_i^\top.$$

Score each candidate by the inverse elliptical bonus

$$b_t(u) = u^\top V_t^{-1}u,$$

and select

$$u_t = \arg\max_{u \in \{u^{(1)},\ldots,u^{(K)}\}} b_t(u).$$

The inverse is important. A score of $u^\top V_t u$ would reward directions already represented in the history, while $u^\top V_t^{-1}u$ rewards directions that are under-covered after ridge regularization.

The hyperparameters are the candidate count $K$ and ridge $\lambda$. Larger $K$ gives the method more opportunity to find under-covered directions, while larger $\lambda$ makes early decisions closer to radius-based iid candidate selection.

Angular Variant

The raw inverse elliptical score can prefer high-radius Gaussian tail samples, especially when $K$ is large. A more conservative version normalizes directions:

$$d(u) = \frac{u}{\|u\| + \epsilon}, \qquad V_t^{\mathrm{dir}} = \lambda I + \sum_{i<t} d(u_i)d(u_i)^\top,$$

then scores candidates by

$$b_t^{\mathrm{dir}}(u) = d(u)^\top (V_t^{\mathrm{dir}})^{-1} d(u).$$

This variant emphasizes angular coverage of the Gaussian support rather than pushing samples toward large radius. It is likely the safer first method for 2D visualizations and for early experiments where tail samples may decode poorly.

Method 3: Kernel-Reweighted Router Sampling

This method defines an explicit reweighted proposal in whitened coordinates:

$$q_t(u) \propto \phi(u)a_t(u),$$

where $\phi(u)$ is the standard Gaussian density and $a_t(u) \in [0,1]$ is a bounded repulsion factor against the history.

A robust kernel repulsion score is

$$D_t(u) = \sum_{i<t}\exp\left(-\frac{\|u-u_i\|^2}{2\sigma^2}\right),$$

with acceptance or importance weight

$$a_t(u) = \epsilon + (1-\epsilon)\exp(-\beta D_t(u)).$$

Here $\sigma$ is the repulsion radius in whitened units, $\beta$ is the repulsion strength, and $\epsilon$ is an acceptance floor. The floor prevents the sampler from stalling when the history already covers much of the high-density region.

Rejection Sampler With SIR Fallback

Because $a_t(u) \le 1$, rejection sampling from the router proposal is direct:

1. sample $u \sim \mathcal N(0,I)$;
2. accept with probability $a_t(u)$;
3. if accepted, set $u_t = u$ and map back to $z_t = \mu_x + L_x u_t$.

This samples from the reweighted proposal exactly up to the normalizing constant when an accepted proposal is found. The practical risk is unstable acceptance rate as $t$ grows. The locked v1 implementation should cap rejection attempts and, if no proposal is accepted, fall back to sampling-importance-resampling from a finite fallback pool using the same weights $a_t(u)$. This keeps the fallback stochastic and aligned with the same reweighted proposal, rather than switching to a deterministic maximum-weight candidate.

Default v1 hyperparameters:

• $\sigma = 0.75$;
• $\beta = 5.0$;
• $\epsilon = 0.02$;
• max_rejection_attempts = 500;
• fallback_candidate_count = 64.

Sampling-Importance-Resampling Variant

Sampling-importance-resampling is useful as the rejection fallback:

1. sample $u^{(1)},\ldots,u^{(K)} \sim \mathcal N(0,I)$;
2. compute weights $w_k = a_t(u^{(k)})$;
3. sample one candidate proportional to $w_k$.

This keeps the fallback bounded without introducing a separate deterministic novelty-reranking behavior. The first standalone synthetic-sequences evaluation should compare only iid router sampling against this kernel-reweighted rejection sampler with SIR fallback, not a separate SIR-only method or a maximum-weight fallback.

First Notebook Scope

The first 2D notebook should compare only these methods:

• iid router sampling;
• inverse elliptical candidate reranking, with both raw and angular scores if useful;
• kernel-reweighted router sampling, using sampling-importance-resampling first and rejection sampling only if acceptance behavior is stable.

The visualizations should show whitened $u$-space samples, the mapped Gaussian ellipse in $z$-space when helpful, and the sequence order of sampled latent states. The main qualitative question is whether the adaptive methods cover high-probability router support with fewer repeats than iid sampling without being pulled into unreasonably low-density tails.

First Synthetic-Sequences Evaluation Scope

The first standalone synthetic-sequences exploration-efficiency script should compare:

• iid: iid router samples $u_t \sim \mathcal N(0,I)$;
• kernel_reweighted: bounded rejection sampling with SIR fallback using the kernel repulsion weights above.

The script should be continuous-only because this method assumes a Gaussian router. It should evaluate strategy coverage by decoding $y_t \sim g(\cdot|x,z_t)$ for each sampled latent and using task-owned strategy attribution, not latent-space distance.