Fedoras are not cool. They are not hip, stylish, or even playfully ironic. You look silly in that thing. Take it off. Unless you’re a hard-boiled detective, a gangster from the ’40s or remember the Great Depression there is just no excuse to commit such an egregious fashion faux pas. We all have a friend who thinks they can pull it off, but tragically can’t. If you don’t have that friend, odds are you might be that friend.

I’m belaboring the point now. Suffice it to say that fedoras are distinctly and irrevocably unfashionable. Luckily there is a general consensus on the issue, and most people steer clear. There is still the occasional person here and there who tries it out. This is a story about them, or, more accurately, our perceptions of those people.

As some readers may have picked up on, I run in some decidedly left-of-left-of-center circles. In these social groups, we have a kind of caricatured image of a certain kind of person in the out-group — libertarians. In our minds, libertarians are just the worst kind of people. They are rude, selfish and arrogant. They are the kind of people whose keyboards are crusted with Cheeto dust. They are the kind of people who sport neckbeards not due to genetic tragedy or affiliation with the Amish community but because they think they look good. And the worst thing about these neckbearded, trenchcoat-wearing, Guy Fawkes aficionados is that they wear fedoras.

It’s not accurate. We know that. It’s a kind of rough intellectual shorthand. They are not incorrect in their views’, they are just bad. It makes disagreements easier and our own beliefs more secure.

I happen to have a number of libertarian friends. While talking with one such friend recently, he referred to a stereotype similar to the one provided above—this time of left-wingers like myself. In the libertarian mind, apparently, leftists are goofy, scraggly-bearded, Che Guevara T-shirt wearing, acne-covered social justice warriors from the depths of Tumblr. And you know what? We also wear fedoras.

Intrigued, I decided to take to the Internet. I found numerous cases of “fedora-ism” all across the web. Conservatives imagine liberals as kale-eating, tax-and-spend, secret socialist fedoristas. Liberals imagine third party people, both Libertarian and Green Party, as awkward, uninformed fedora-wearers.

Bizarre, right? How and why does this happen?

Personally, I think it just makes things easier. So absolute is the social taboo associated with fedoras that anyone who wears one is immediately delegitimized and subject to criticism. We can write off their ideas and sit a little more comfortably in our own opinions. It helps to know that those whose opinions differ from ours don’t just disagree with us, they are completely and hopelessly wrong about everything.

This is comforting. It is. It also seriously cripples any attempts at meaningful political discourse. It may be hard, but we need our imaginary libertarians, liberals, conservatives, etc., to remove their fedoras. We need to take each other seriously. We don’t have to agree, but so long as we keep arguing with these caricatures, we will never really get to the bottom of bigger questions.

And so I’m asking you, take off the fedora. Figuratively and literally, please.