Convolutional neural network (CNN)-based image classifiers can accurately classify histological type and ER/PR/HER2 status of breast cancer patients using only H&E slides. Here, ESR1 is linked to breast carcinoma.