Brainstorming framing and scope: how should paper's contributions be framed?

• Papers in machine learning are often franed as "our method vs prior methods"

  • Reviewers are only briefly scanning a paper, and are expecting tables or figures that

• This template doesn't fit our research very well…

  • In the future, after we've developed these initial ideas, we plan to apply them towards supporting active directed (potentially analogical) exploration / search in post-training or test-time scaling.
  • At that stage, we could compare to some baselines for post-training / etc (e.g., GRPO-style passive exploration, on-policy pass@k, etc.)
  • However, at this stage, we are mainly formulating a new type of problem and evaluating methods to solve the problem. We empirically evaluate several methodologies of our method with a number of different knobs and variants. But beyond that, there isn't necessarily a natural "prior art" baseline.

Key Questions:

• What is the story we are trying to tell?
• What are the key claims we want to make? How do we want to demonstrate them?
• For pre-trained models: which set of models should we include? Does it matter to have a variet, or is one or two models of modest size enough to demonstrate the key claims?
  • E.g.: Qwen-2.5-0.5B, Qwen-2.5-1.5B? Qwen-3.5-0.8B, Qwen-3.5-2B? Pythia series?
• Is model size of important significance? Should we sweep across model sizes, or is that wasted compute (focus should be elsewhere)?
• Should we put effort into including a more theoretical formulation + simple theoretical results on a toy model?
  • A possibility is to study a toy model where $\exists f_1, \ldots, f_K: X \to Y$ and $p(y|x) = \sum_{i=1}^{K} p_i \delta_{y = f_i(x)}$. Can we show that (for some choice $\beta$?) that our procedure can identify $f_1, \ldots, f_K$ in some sense?
  • Having a theoretical component to the story can strengthen the framing of the paper as "formulating a new type of problem and initiating its exploration", and can alleviate criticisms about "synthetic experiments".
• Decide on terminology to use: "factorization", "strategy-disentanglement", "excess reconstruction", "$p(y|x)$-normalized reconstruction", "$(z - x)$-excess reconstruction loss", etc…