What do you do when someone reacts rudely because your baby is crying in public?

I look at them rudely in return. Why react rudely to an innocent child's crying? There was one time, a man was murmuring with his girlfriend just because my baby was noisy. I asked him, "Got a problem?"