We introduce an expert-guided XAI framework to improve the transparency and trustworthiness of deep learning models for insulin titration in diabetes management. This evidence concerns the gene INS and diabetes mellitus.