I Have Covered All the Bases Here: Interpreting Reasoning Features in Large Language Models via Sparse Autoencoders

Citation

• Authors: Andrey Galichin, Alexey Dontsov, Polina Druzhinina, Anton Razzhigaev, Oleg Y. Rogov, Elena Tutubalina, and Ivan Oseledets
• Year: 2025
• Venue: arXiv preprint
• arXiv: arXiv:2503.18878
• URL: https://arxiv.org/abs/2503.18878
• Citation count: 4 (ResearchGate, retrieved 2026-04-20)

One-Line Takeaway

This paper uses sparse autoencoders to isolate internal features associated with reasoning behavior in reasoning-tuned LLMs, then shows that steering those features changes both reasoning quality and trace length.

Technical Core

• Train a sparse autoencoder on a layer activation $h$ of a reasoning LLM so that $$h \approx W_{\mathrm{dec}} f(h),$$ where the feature vector $f(h)$ is sparse and is meant to expose more interpretable factors than raw dense activations.
• Build a reasoning vocabulary of words that are unusually common in model "thinking traces."
• Define ReasonScore to rank SAE features by how strongly and broadly they activate on reasoning-vocabulary tokens and their surrounding context.
• Use feature steering by adding a scaled decoder direction back into the model activations: $$h' = h + \gamma \, f_i^{\max} W_{\mathrm{dec}, i},$$ where feature $i$ is an SAE feature chosen for steering.

Method

• Train SAEs on internal activations of a reasoning model from the DeepSeek-R1 family.
• Use ReasonScore to shortlist candidate reasoning features.
• Interpret those features manually and with an LLM-based automatic interpretation pipeline.
• Run steering experiments on selected features and evaluate both benchmark accuracy and reasoning-trace length.
• Use model-diffing and cross-layer / cross-architecture checks to ask when these features emerge and whether they generalize.

Empirical Evaluation

• The paper identifies a core set of reasoning-related sparse features corresponding to behavioral motifs like uncertainty, exploration, and reflection.
• Steering some of those features improves benchmark performance on reasoning-heavy evaluations such as AIME, MATH, and GPQA.
• Steering also changes reasoning-trace length in systematic ways, including at least one feature that shortens traces without large performance loss.
• The reasoning features appear mainly after reasoning-specific fine-tuning rather than being fully present in the base model already, and analogous features are found across multiple layers and model families.

Assumptions And Limits

The approach depends on the chosen SAE layer, the selected reasoning vocabulary, and the interpretability of sparse features after factorization. The discovered features are post hoc and local to a trained model family rather than a clean generative latent-variable factorization. The paper also studies reasoning-tuned models, so it does not answer whether the same structure exists cleanly in ordinary pretrained base models.

Relevance To This Project

This paper is the strongest nearby example of a non-VAE route to latent reasoning structure. Instead of training a new bottlenecked model, it asks whether reasoning variables can be found inside a model that already exists.

That makes it very relevant to the repo's interpretability motivation:

• it suggests that latent reasoning structure may already live in hidden representations before any explicit factorization is learned;
• it offers a post hoc alternative to the repo's generative-factorization approach;
• it also sharpens the project's distinct bet: learned latent factorization may give cleaner routing and intervention handles than post hoc feature extraction alone.

Comparison Notes

Tan et al. (2024) exposes interpretation through supervised human concepts. This paper instead discovers sparse internal features after training. Relative to this repo, sparse-autoencoder reasoning features are closer to "mechanistic probes plus steering," while the repo aims for a reusable latent variable inside the generative model itself.

Related Notes

• Interpreting Pretrained Language Models via Concept Bottlenecks
• Abstract Strategy vs Solution
• Contributions
• Challenges of Learning a Latent Variable Factorization of Pre-trained Language Models