Manipulating attentional priority creates a trade-off between memory and sensory representations in human visual cortex

People often remember visual information over brief delays while actively engaging with ongoing inputs from the surrounding visual environment. Depending on the situation, one might prioritize mnemonic contents (i.e., remembering details of a past event), or preferentially attend sensory inputs (i.e., minding traffic while crossing a street). Previous fMRI work has shown that early sensory regions can simultaneously represent both mnemonic and passively viewed sensory information. Here we test the limits of such simultaneity by manipulating attention towards sensory distractors during a working memory task performed by human subjects during fMRI scanning. Participants remembered the orientation of a target grating while a distractor grating was shown during the middle portion of the memory delay. Critically, there were several subtle changes in the contrast and the orientation of the distractor, and participants were cued to either ignore the distractor, detect a change in contrast, or detect a change in orientation. Despite sensory stimulation being matched in all three conditions, the fidelity of memory representations in early visual cortex was highest when the distractor was ignored, intermediate when participants attended distractor contrast, and lowest when participants attended the orientation of the distractor during the delay. In contrast, the fidelity of distractor representations was lowest when ignoring the distractor, intermediate when attending distractor-contrast, and highest when attending distractor-orientation. These data suggest a trade-off in early sensory representations when engaging top-down feedback to attend both seen and remembered features and may partially explain memory failures that occur when subjects are distracted by external events.