I see my baby everyday, but I pump at work. I express my milk and store it in bottles for her feed the next day. On Fridays, I express it into milk bags for it to be frozen.
My baby can handle both frozen and chilled milk. In your case, I've read that chilled milk can be stored safely for up to 8 days, so it should be fine for you to go without freezing the milk.
http://www.llli.org/faq/milkstorage.html