What and where manifolds emerge and align with perception in deep neural network models of sound localization

Whether the auditory cortex has parallel pathways for sound identification ("what") and localization ("where"), and whether it contains a map of auditory space, is debated. Here, we examined the low-dimensional structure (manifold) of "what" and "where" representations in deep neural network models of sound localization. Unexpectedly, models trained for "where" learned untangled "what" manifolds, including voice type, reverberation, and spectral detail. The distribution of "what" manifolds was not random, but geometrically organized by spectral similarity. The separability and distance of both "what" and "where" manifolds were aligned with human behavior. "What" also determined whether "where" manifolds organized into a map: maps emerged when "what" contained localization cues that were topographically organized. However, forming a spatial map reduced localization accuracy in both models and human listeners. Together, object manifolds reveal learned task-irrelevant attributes of object that are ignored when measuring task performance alone, and task-optimized neural networks can provide insights into brain and behavior, not just replicate them.