A fine-grained understanding of functional selectivity in human cortex is fundamental for elucidating how visual information is processed and represented in the brain. Classical work utilizing designed experiments has uncovered a number of category-selective regions. However, this approach of cognitive neuroscience requires preconceived hypotheses about the categories. More data-driven approaches that employ rich sets of natural stimuli and simple encoding models have so far lacked the capacity to capture the complexity of neural responses under naturalistic conditions. We propose a novel in silico approach for data-driven discovery of novel category-selectivity hypotheses. We use an encoder-decoder transformer that leverages a brain-region to image-feature cross-attention mechanism, designed to non-linearly align high-dimensional deep network features with cortical responses. Unlike earlier encoding models that apply linear regression to deep-network activations, our architecture learns to flexibly map stimulus features to semantic patterns of fMRI activity in an end-to-end manner. We introduce a method that leverages diffusion-based image generative models and large image datasets to synthesize and select images that maximally activate different cortical parcels. Our method reveals regions with complex compositional selectivity involving diverse semantic concepts. In silico mapping of categorical selectivity provides a data-driven approach for generating novel visual-selectivity hypotheses about the human brain---to be confirmed in future fMRI studies.