Main libertarian objections to the Fair Tax:
1. The prebate would start a new welfare entitlement.
2. The transition would redistribute from savers to borrowers.
3. There is a danger of getting BOTH an income AND a consumption tax.
4. Advocates disingenuously quote a 23% rate when it is actually 30%.
5. Advocates use protectionist rhetoric to sway populists.
Also well worth a read is Jason Gonella's open letter to Johnson, which covers some other issues.
And two pieces on the "Fair" Tax by LP presidential nomination candidate R. Lee Wrights (here and here).
And finally, while I don't by any means claim to be "the father of libertarian opposition to the 'Fair' Tax,'" I can claim to have done a bit of writing on it long before it became a football in the Libertarian Party's 2012 presidential nomination process -- see here and here.