What do you do when someone reacts rudely because your baby is crying in public?

I will just say "excuse me. he is my child, mind your own business" and walk away incase they do something or continue to argue with us.