The landscape of cancer treatment has been transformed over the past decade by the success of immune-targeting therapies. However, despite sipuleucel-T being the first-ever approved vaccine for cancer and the first immunotherapy licensed for prostate cancer in 2010, immunotherapy has since seen limited success in the treatment of prostate cancer. The tumour microenvironment of prostate cancer presents particular barriers for immunotherapy. Moreover, prostate cancer is distinguished by being one of only two solid tumours where increased T cell-infiltration correlates with a poorer, rather than improved, outlook. Here, we discuss the specific aspects of the prostate cancer microenvironment that converge to create a challenging microenvironment, including myeloid-derived immune cells and cancer-associated fibroblasts. By exploring the immune microenvironment of defined molecular subgroups of prostate cancer, we propose an immunogenomic subtyping approach to single-agent and combination immune-targeting strategies that could lead to improved outcomes in prostate cancer treatment.