Sounds like a tricky situation to be in.
How is her interaction with your child so far? Sometimes, I feel that grandparents behave differently when they are with their grandchildren as compared to when they are with their own children.
Regardless, I personally feel that spending a few hours a week with the grandparents is unlikely that have any major negative impact on the child. Especially since you will be present during the visit and could explain to your child if required. Therefore, I would think that you can consider increasing the number of visits to once a week. I think, in general, it is still better if your child gets to know her grandmother. I feel that children who grew up with a healthy exposure to their grandparents are more compassionate and empathetic. But these are just my personal opinion, what matters most is really what you are comfortable with.