Yeah I think that is the better option honestly, I think your better of to switch to almond milk or soy milk.
People have seen huge results when they cut out milk, I was in the process of cutting it out but when I started bulking everyone told me it was a great way to add extra calories and fat so back in it went.