You are the employer so feel free to set boundaries and do what you're most comfortable with. When you want to take care of your child, you should do it. For my helper her main task is to take care of the toddler because there are only few chores to do (light cleaning, laundry, sometimes cooking), and when she's busy doing her chores I'll step in to take care of the toddler. When I feel like it, I take small breaks during my wfh and help her with feeding, bathing, entertaining etc, but by default these are her job. If you offer help I don't think the helper or your husband have any reason to object? And just because she's taking care of your child doesn't mean she can replace the role of a mother, so don't worry about that :)
Ha