Drinking Fresh Milk While Breastfeeding

I realised that on days I drink milk, my baby tends to spit up/ vomit more than usual. Is there really a link to this? I am fully breastfeeding.