Just make sure that however short the time you have for your LO before he/she sleeps, is a good quality time. Do things with him/her that only you can do e.g. latch, mirror games, see-saw, sing a special song, read a book (different from all the books that the caregiver reads), etc,
It’s difficult and heartbreaking at first but it does get better over time.