Are there any disadvantages of not to letting baby have a pacifier? Like affect their brain development?

Don't think there are any disadvantages. It's also a matter of choice whether you choose to give baby the pacifier.