Nonlinear ICA Using Auxiliary Variables and Generalized Contrastive Learning

Citation

• Authors: Aapo Hyvarinen, Hiroaki Sasaki, Richard E. Turner
• Year: 2019
• Venue: AISTATS 2019 / Proceedings of Machine Learning Research 89, 859-868
• arXiv: arXiv:1805.08651
• URL: https://proceedings.mlr.press/v89/hyvarinen19a.html

One-Line Takeaway

Hyvarinen, Sasaki, and Turner show that nonlinear ICA becomes identifiable once the latent sources are conditionally independent given an observed auxiliary variable $u$, and they turn that identifiability argument into a practical contrastive-learning estimator.

Technical Summary

• Setup: the observations satisfy $$x = f(s),$$ where $f$ is smooth and invertible and the latent sources $s_1,\ldots,s_n$ are conditionally independent given an observed auxiliary variable $u$: $$\log p(s \mid u) = \sum_{i=1}^n q_i(s_i, u).$$
• General-case identifiability: if the conditional source laws vary with $u$ in a sufficiently rich way and the learned feature map $h(x)$ is invertible and smooth, then the learned coordinates recover the sources up to component-wise invertible scalar transformations.
• Variability condition: the paper's key nontrivial assumption is that $u$ changes the source conditionals strongly enough that certain first- and second-derivative feature vectors built from $q_i(\cdot, u)$ are linearly independent across several values of $u$.
• Conditionally exponential case: if $$p(s_i \mid u) = \frac{Q_i(s_i)}{Z_i(u)} \exp\!\left(\sum_{j=1}^k \tilde q_{ij}(s_i)\lambda_{ij}(u)\right),$$ then the estimator recovers sufficient statistics of the sources only up to an additional linear transformation. For low-order conditional exponential families, this linear indeterminacy is unavoidable.

Method

• Build augmented data pairs $\tilde x = (x, u)$ and randomized negatives $\tilde x^\ast = (x, u^\ast)$, where $u^\ast$ is permuted independently of $x$.
• Train a nonlinear logistic regressor of the form $$r(x, u) = \sum_i \psi_i(h_i(x), u)$$ to discriminate true pairs from randomized pairs.
• Under infinite data and universal approximation assumptions, the learned hidden units $h_i(x)$ consistently estimate the nonlinear ICA solution allowed by the corresponding theorem regime.

Empirical Evaluation

• Non-conditionally exponential experiment: the paper generates five nonstationary sources on a $2$D grid by modulating Laplacian variables with spatially varying scales, mixes them with a random three-layer neural network, and uses the spatial index pair as $u$.
• Conditionally exponential experiment: the paper generates segmented nonstationary time-series sources, uses the segment index as $u$, and evaluates recovery up to the unresolved sign / linear ambiguity of that regime.
• Headline result: the proposed method performs similarly to TCL, while supporting a broader and more explicit identifiability framework.

Assumptions And Limits

The result depends on observed side information $u$ that is not just a deterministic function of $x$, on invertibility / smoothness assumptions for the generative and learned maps, and on a strong variability condition for how $u$ modulates the source laws. When the conditional source family is too simple, especially low-order conditional exponential families, a linear ambiguity remains.

Relevance To This Project

This paper is highly relevant to the repo's strategy-identification agenda because it gives a clean theorem for when extra observed structure restores latent identifiability. In x -> z -> y terms, it says that a latent strategy variable can become recoverable once an auxiliary observable changes the latent distribution in a sufficiently rich way. That is the same structural lesson behind using time, metadata, interventions, or repeated contexts to identify latent reasoning modes.

Comparison Notes

Compared with Jiang and Tanner (1999), the auxiliary variable is not a covariate-dependent gate over experts but an observed variable that modulates the latent source law directly. Compared with Khemakhem et al. (2020), this paper works in the nonlinear-ICA setting and shows the same basic principle more directly: unsupervised latent-variable models are not identifiable from marginals alone, but conditioning structure can sharply reduce the ambiguity.

Related Notes

• Formalizing Strategy Identification with Expressive Generator Classes
• Background on identification and learning of mixture distributions
• Variational Autoencoders and Nonlinear ICA: A Unifying Framework