When is the good time to bottle feed breast milk? Will LO reject breast feeding after that?

I started bottle feeding ard 3.5 mths as lo is attending ifc. However, this doesn't stops him from latching. He must latch whenever he is home. Tried bottle feeding him when he's home but he rejects all the way. Maybe it's the smell of mummy. Don't bottle feed unless necessary. Jia you