I was in your shoes 3 years ago. The pay was good but the boss was s***. No time to get married, no time to have kids. You have to decide when enough is enough and walk away from all this. I have no regrets quitting my job although I did tell my hubby that perhaps in the future if I hear my boss is gone, I might decide to go back there to work.
Being a sahm means loss of income from one source. But if you and hubby work out your finances and feel that it's ok then that's fine.