You should talk to your husband about this, and no matter what people say or not, do not take your children to their home. See, what I believe is, if something is taking away the mental peace of the family as a whole, why to do that just to please others or avoid questions. And if I were you, I would have frankly told after the repetitive incidents that my kids do not like to come over because of your behaviour with them. I think you too should do the same or just stay put at your place and be happy.