It depends on how you and your husband feels.
As i am staying with in laws under one big roof and we do not have a spare room OR a place which our kid can bunk in with.
Usually we do it when the kid is in deep sleep. We will def be slient in doing it. but priority is always the kid. whenever he started fussing or crying, we would stop and i would have to settle him before continuing.
Sometimes my husband feels it's fun that he will have to 'speed up' whenever he sensed that our kid is gg into a full blown cry soon