Breast changes during pregnancy can be really all over the place, honestly, so some shrinking after that initial growth in the first trimester isn't uncommon, but it's definitely worth flagging to your gynae or PD just to be on the safe side. I'd bring it up at your next check-up because they can assess whether it's normal fluctuation or if there's something they want to monitor, especially since you're halfway through already. In the meantime, try not to stress too much as pregnancy hormones do weird things to our bodies, but your doctor will give you the clearest answer.