Normalizing Flows (NFs) are universal density estimators based on Neuronal
Networks. However, this universality is limited: the density’s support needs to
be diffeomorphic to a Euclidean space. In this paper, we propose a novel method
to overcome this limitation without sacrificing universality. The proposed
method inflates the data manifold by adding noise in the normal space, trains
an NF on this inflated manifold, and, finally, deflates the learned density.
Our main result provides sufficient conditions on the manifold and the specific
choice of noise under which the corresponding estimator is exact. Our method
has the same computational complexity as NFs and does not require computing an
inverse flow. We also show that, if the embedding dimension is much larger than
the manifold dimension, noise in the normal space can be well approximated by
Gaussian noise. This allows to use our method for approximating arbitrary
densities on non-flat manifolds provided that the manifold dimension is known.