New member here. Long time "lurker".
Let's not be quick to hammer the installer just yet! Is your drywaller experienced with ICF? Reason I ask, is because all of us who have been in construction a long time know that the "new kid" on the block takes the hit for everyone! This is why on occasion, other trades will "pad" their bids because they are unfamiliar with the ICF method. Those of you in construction know that every trade blames the other for issues.
There are situations that can cause walls to be out of plumb! Stuff happens. If it is grossly out of plumb, then there is a problem. Might I suggest a meeting between you, the ICF installer, and the drywall installer and sort it out. Usually when you have all pertinent parties involved face to face, big things get little really quick!
Not saying this is the case here, but just my experience!
BTW, you paid the installer for walls installed correctly. You should not incur and additional charges to make the walls plumb! "Furring out" while undesireable and avoidable with proper pre pour due dilligence does become necessary sometimes.
Good Luck! Even though there are always going to be issues in construction, you will forget them once the house is done and you are enjoying a "high performance" home!