When do you stop using an infant insert? I am using the ergo infant insert

If I remember correctly, I stopped using the insert when my baby was about 4 months. One thing is my baby dislike it... So I hardly use it also.