It depends on individual. Usually 2nd to 3rd trimester, but for some people there is no increase in size. But don’t worry, this does not affect milk supply :)
My first pregnancy only swollen throughout then bigger after I delivered. 2nd pregnancy It got bigger during first trimester towards the 2nd.