breast milk and formula

Hi mummies! My nb is just 5 days old but i have been feeding both breastmilk and formular, but if i want to swap to full on latch on will it be too late? as i was told that baby will not be close to you if feed too much formula