Does it look like this?
(taken from: http://www.skinsight.com/infant/supernumeraryNipple.htm)
According to the website, Supernumerary Nipple usually appears as a small tan or brown freckle or mole that is present somewhere on the abdomen, below the nipple.
If it is Supernumerary Nipple, it probably will not go away but could be removed surgically if required.