annoying mother & mil

Hi . Just wanna rant here . I have a 4 years old son , and I've always been calling him " baby " . So even in public I'll go " baby come , let's go home " " baby , eat ok ? " And my mum and mil just came to tell me that I should stop . They said it's not good for his development if I always instill in him that he's still a baby . Any thoughts on this ? I'm so mad .