Hi dear, this could be because of a few reasons like he has had enough milk from one side already, or he could be colicky, or some babies have a breast preference, or he has pain in his ear.
This has happened to me as well. So I started to alternate between 10 mins feed to see if he refuses while still hungry. After two days, he started feeding from the other one too. I think it was because I wasn’t evenly distributing time on both.
I also feel like with engorged breasts, the baby fills up quickly so you are left with this issue.
If you try this and it doesn’t work, I would suggest taking your baby for a checkup. Take care.