Motherhood is tough and it can make us think crazy things. I recommend take a step back and remember the man you married and the man that you love. Sometimes our emotions get the best of us and we let it ride because we are so overwhelmed by the chaos of life. I suggest you catch him on a good day, and talk to him.
The guy is loyal, give him the benefit of the doubt.