Holding others baby

What's wrong carrying over other's baby from a guy? My husband said common sense it's inappropriate for me to do that. Really?