I think it depends. Some people struggle even with larger nipples if baby don’t latch well. I have both smaller and inverted nipple on one side and a normal sized protruding nipple.
Initial stage will be harder but once baby latches well, it’s all good. I always offer the inverted side then the normal side else baby won’t “work” for it after having the easier boob.