Do you put water in bottle warmer and then put the milk bottle to warm it?

Yes, you need to put water in the bottle warmer. In case you are using an Avent one like me, here is a link to how to use it properly: http://www.philips.com.sg/c-p/SCF355_00/avent-fast-bottle-warmer#see-all-benefits