Be honest -- money is often a sensitive and difficult thing to deal with in a relationship and if it bothers you too much, just say no. Alternatively, you can suggest that you will take on the duty but only for household related stuff (groceries, children) as you have a better understanding of these things and not his allowance etc. I'm sure he will understand.
Read more