Husband or me at fault?

Husband doesn't like the idea & don't allow me to bring baby to visit my grandparents due to the virus. He feels thats my grandparents should come over instead. I explain to husband that I needed some 'me time' & 'personal space' but he doesn't understand. He even feels that I am not thinking right, why am I putting baby to the risk. He allows me to go back to grandparent's place w/o baby. How can he separate us like that? The last time I brought baby back to grandparents place was about a month ago. I just wanna seek outsider's opinion. Who is at fault?