Parents coming over uninvited

Hi everyone. I am in a difficult situation. We gave set of keys to both our parents in case anything of emergency. With that, my father comes over almost everyday uninvited. I am okay with it but my spouse on the other hand thinks its rude and disrespectful. He wanted to tell my dad off but i stopped him as you know parents get sensitive as they age. My spouse has repeatedly say he is going to tell him off but i do not want to ruin the already perfect relationship we have now as family/in laws. I know i am wrong for stopping my spouse as this is his house too. But my dad only wanted to see his grandchildren. Just thats its abit too excessive😔 What should i do?