A judge has ruled a Montpelier man won't be able to withdraw from a plea agreement for his sexual assault cases, but he will get a new attorney.