My LO doesn't like me.

He cries and throw tantrums most of the times when I carry him. whenever grandma or his father carries him he is calmer. Especially now when he is sick I don't want him to cry too much and I keep worrying that he is not breathing. Most of the time my mother handles him so I understand. They keep telling me it's because they handle him most of the time that is why but no matter how I try it doesn't seem to work.