Perhaps your little one miss your attention, love and affection. You may want to well balance your time with your toddler, do activities together over the weekend or free time. If your nanny's home you may ask her to do other stuff while you care for the little one so you can divert that attention more to your toddler. Hopefully it will get better.