By examining 103 BRCA1/2 and PALB2 mutation negative familial and 75 young breast cancer cases, together with 128 geographically matched healthy female controls, we show that the frequency of rare CNVs is increased in cases when compared to controls and that the genes disrupted in individuals of specifically the two case groups are closely related to estrogen signaling and TP53 centered tumor suppressor network. Here, PALB2 is linked to breast cancer.