This phase I/II trial studies the side effects and best dose of HCQ when given together with palbociclib and letrozole before surgery in treating participants with estrogen receptor positive, HER2 negative breast cancer. The gene discussed is ESR1; the disease is breast cancer.