As long as the ice is still cold and can keep the milk cold, it should not be a problem.
Use more ice packs so that ice packs keep each other cool for longer periods! What I do is I will have 2 500g ice packs ($2 each, Daiso) for the front and back then squeeze in little cuttable 50g ice packs ($2 for 6, Daiso) in between the milk. Another alternative is to get the fridge-to-go that can last you for 12 hours. I couldn't convince myself to spend the $79.90 and it's bulky and heavy.
Something like this would do as well. Just have more ice packs and see what works for you.