Should I be worried that my son's the smallest in his class? He's 5

No need to worry. Boys can suddenly grow up a lot during puberty period and besides it is still too early to tell if he will be small. Even if he is going to be we still should accept the fact and most importantly is he is healthy :)