Manifold geometry underlies a unified code for category and category-independent features

In everyday vision, animals routinely extract from the same visual stimulus both object identity and continuous identity-independent variables such as position and size. It has been shown that linear decoding performance of both kinds of information increases along the ventral stream, suggesting that inferior temporal cortex may be implementing a joint code for object category and category-independent features. A central open question is whether such a code can indeed exist within a single representation and, if so, what geometric properties enable it. Here, we show that convolutional neural networks can develop such a code. We then derive a theory of regression on category manifolds, identifying the key manifold-geometry measures that enable accurate readout of category-independent features, and showing how they can be optimized while preserving manifold properties known to support classification performance. We further characterize how common experimental constraints, such as subsampling neural units and using a limited number of categories, affect the empirical estimation of regression performance. Our findings thus provide a principled understanding of the geometry underlying joint codes and yield testable predictions for future neural recordings probing the joint-code hypothesis in the ventral stream.