Your baby sounds like he's doing fine if his diapers are wet and he's gaining weight, so I wouldn't worry too much. Every baby is different and some are just more efficient feeders or need less than others, so as long as he's meeting those milestones there's really no need to push more milk into him. That said, it might be good to bring him to your PD for a check-up just to be safe and get their reassurance, especially since you're concerned enough to ask.