We reconsider the study of the eigenvectors of a random matrix, to better understand the relation between localization and eigenvalue statistics. Traditionally, the requirement of base invariance has lead to the conclusion that invariant models describe extended systems. We show that deviations of the eigenvalue statistics from the Wigner-Dyson universality reflects itself on the eigenvector distribution. In particular, gaps in the eigenvalue density spontaneously break the U(N) symmetry to a smaller one. Models with log-normal weight, such as those emerging in Chern-Simons and ABJM theories, break the U(N) in a critical way, resulting into a multi-fractal eigenvector statistics. These results pave the way to the exploration of localization problems using random matrices via the study of new classes of observables and potentially to novel, interdisciplinary, applications of matrixmodels.