One line I hear frequently in internal libertarian movement debate is that it's entirely possible to be both a bigot and a libertarian. That is, one could conceivably have an aversion to some group (racial, gender/sexual minority, whatever) without advocating for the initiation of force against that group.
True as far as it goes, I guess. And it could also be said that it's entirely possible to be a libertarian and also to really like goat feces, diesel fuel, and fire.
But I'm betting that if there are media and public inquiries to the Libertarian Party after some idiot rolls around in a mixture of goat feces and diesel fuel, then sets himself on fire in his front yard while screaming "LIBERTARIAN! LIBERTARIAN!" the response is going to be "yeah, that's not us."