When you say aggressive, to what extent? Has he actually hurt another child? I would recommend signing him up for a sports activity where he can play rough but with control and rules. Sports can instill control and discipline, give him a space to unleash his energy but with boundaries. Perhaps rugby or a form of martial arts? I've heard so many stories of "aggressive" kids doing well in sports and learning to control their energies.
As for the other parents, they are just being protective, albeit a bit too much, so just ignore them and focus on your son. Good luck!