Oh dear, I can understand how worrying this must feel for you, especially when it is not the usual kind of pain. Since the doctor has already advised you to go back if there is fever, please do monitor your temperature closely and do not hesitate to return even if something just does not feel right to you, a mother's instinct is very important. It is always better to get it checked earlier rather than wait, so if the swelling around the areola is not improving or you notice any other changes, please do go back to see your doctor soon. Sending you warm wishes and hope you feel better very quickly, sayang.