Did you have an agreement prior to having your own child? If so, you can go back and remind him about what you have agreed upon when it comes to spending time with his kids. You should come up with an agreement moving forward so you won't have any conflict on how he divides his time with the kids.