Here, we combine deep learning image models from digitalized histopathology slides with transcriptomic analyses to uncover molecular and morphological profiles associated with hormone receptor status and genomic subtypes of breast cancer. Here, NR4A1 is linked to breast carcinoma.