Semiparametric mixtures of nonparametric regressions

Citation

• Authors: Sijia Xiang, Weixin Yao
• Year: 2018
• Venue: Annals of the Institute of Statistical Mathematics 70(1), 131-154
• arXiv: None
• URL: https://doi.org/10.1007/s10463-016-0584-7

One-Line Takeaway

Xiang and Yao study a mixture-regression model with constant mixing proportions and variances but smooth component mean functions, and show that this semiparametric restriction buys both identifiability and substantially better estimation efficiency than a fully nonparametric conditional mixture.

Technical Summary

• Setup: for fixed $k$ and univariate covariate $X$, $$Y \mid X = x \sim \sum_{j=1}^k \pi_j \,\phi\!\left(Y \mid m_j(x), \sigma_j^2\right),$$ where the component mean functions $m_j(x)$ are smooth, but the mixing weights $\pi_j$ and variances $\sigma_j^2$ are global constants.
• Identifiability: the model is identifiable when the $m_j$ are differentiable, the covariate domain is an interval, and either the component variances are distinct or any equal-variance components differ through their mean functions or derivatives: $$|m_i(x)-m_j(x)| + |m_i'(x)-m_j'(x)| \neq 0 \quad \text{for all } x.$$
• The paper positions this model between ordinary mixture-of-linear-regressions models and fully nonparametric mixtures with $\pi_j(x)$ and $\sigma_j^2(x)$ also varying in $x$.

Method

• Spline initialization: approximate each $m_j(x)$ with a B-spline basis and run an ordinary EM fit for a parametric mixture regression as an initialization.
• One-step backfitting:
  1. Run a modified EM-type algorithm for the fully nonparametric mixture model to get pilot estimates of $\pi_j(x)$, $m_j(x)$, and $\sigma_j^2(x)$ on a grid.
  2. Given the pilot curves, run a standard EM update for the global parameters $(\pi_j, \sigma_j^2)$.
  3. Given those global estimates, run another modified EM-type update for the nonparametric functions $m_j(x)$.
• Global EM-type algorithm (GEM): a cheaper iterative variant that updates the global parameters and nonparametric component functions jointly while preserving an asymptotic ascent property.

Theoretical Result

• Global parameters: the one-step backfitting estimator for $\beta = (\pi^\top, (\sigma^2)^\top)^\top$ is asymptotically normal at the parametric $\sqrt{n}$ rate.
• Nonparametric functions: $\hat m(x)$ has the same first-order asymptotic behavior it would have if the global parameters were known in advance, with the usual kernel-regression $\sqrt{nh}$ scaling after bias correction.
• Algorithmic monotonicity: the modified EM updates and GEM satisfy asymptotic ascent properties.
• Inference: the paper proposes a generalized likelihood ratio test for the null that mixing proportions and variances are constant, and proves an asymptotic $\chi^2$ law under the null.

Empirical Evaluation

• Simulation studies compare the proposed semiparametric model against ordinary mixture linear regression and the fully nonparametric mixture-regression baseline of Huang et al. (2013).
• Real-data examples use US house price index data and nitrous-oxide concentration data as functions of a single covariate.
• Headline result: when the constant-weight / constant-variance assumption is reasonable, the proposed model improves estimation and mean-squared prediction error relative to the fully nonparametric alternative while still capturing nonlinear component means.

Assumptions And Limits

The theory is developed for fixed $k$ and univariate covariates, with smooth component curves, kernel / bandwidth conditions, positive-definite information matrices, and known component count. If the mixing proportions or variances genuinely vary with $x$, the model is misspecified and the more flexible fully nonparametric alternative is the better reference class.

Relevance To This Project

This paper is a useful conditional-mixture reference for the repo because it isolates a particularly practical middle ground: keep the latent routing distribution simple enough to estimate well, but let the component response functions stay flexible. In x -> z -> y terms, it is a case where the router is low-dimensional and global while the strategy-conditioned generator is nonparametric, which is a plausible design pattern for strategy-latent models that need both interpretability and flexibility.

Comparison Notes

Compared with Jiang and Tanner (1999), the router here does not depend on $x$ and the main flexibility is pushed into the component mean functions. Compared with the fully nonparametric mixture-regression model of Huang et al. (2013), Xiang and Yao show how semiparametric restrictions can improve efficiency without collapsing back to a rigid linear-regression mixture.

Related Notes

• Formalizing Strategy Identification with Expressive Generator Classes
• Background on identification and learning of mixture distributions
• Hierarchical Mixtures-of-Experts for Exponential Family Regression Models: Approximation and Maximum Likelihood Estimation