Should I be worried that my son's the smallest in his class? He's 5

Nope. They will grow and spike without you realising. By then they would have really shoot up in height and double up in size.