If you open any centrist or conservative publication of repute, you would read that democracy, expression and freedom are under threat. Not by authoritarian governments, but by “cancel culture.” The phrase is a vague, nebulous force that is very rarely defined by those who frequently deride it.

In the tumult of the past 12 months, these voices — supposedly silenced — have only gotten louder. Most infamously, in a letter in Harper’s Magazine, 153 signatories, ranging from the leftist thinker Noam Chomsky to right-wing intellectuals like Bari Weiss, hyperbolically suggested that “cultural institutions are facing a moment of trial.” Moreover, the phenomenon has not been restricted to the United States, with voices from South Africa and India to myself over two years ago criticizing a perceived absence of intellectual diversity.

Frustratingly, almost all of these laments about cancel culture and intellectual diversity resort to formulaic arguments focusing on the importance of the freedom of expression. Such proceduralist arguments are both incorrect and detract from any genuine debate. Any honest discussion would require an acknowledgement that this is not an issue of freedom of speech. The right to speak is not the same as the right to be heard, much less the right to be heard on a particular platform. A real freedom of speech controversy involves imprisonment, not a removal from Twitter.

Instead, the debate about cancel culture is essentially a debate about cultural priorities. As conservative writer Ross Douthat — a frequent critic of supposed “cancel culture” acknowledged, “all cultures cancel; the question is for what, how widely and through what means.” By definition, every institution that publishes, for example, makes decisions about which voices to center, which voices to include and which voices to exclude. If those decisions increasingly prioritize lived experience, genuine insight and prevention of harm at the cost of some vague notion of intellectual diversity, then that represents social improvement, not a threat to freedom of speech.

There is a tendency to criticize the use of principles of harm in such decisions, for definitions of harm may be inconsistent across a variety of contexts. But the quest for consistency is inherently problematic, as it suggests that there can be a single, universal, precise conception of harm which applies equally to all contexts. Almost by definition, harm is context dependent and operates differently in each setting. An article which may be sensitive in one context may actively cause harm in another.

Moreover, such evaluations of harm can only occur when decision makers come from diverse backgrounds and can approach any such decision through the unique nature of their lived experiences. When critics of cancel culture — usually from privileged backgrounds — deride such claims of harm by marginalized groups, they not only deny the lived experiences of the individuals making those claims, but also suggest that they are more qualified to make the decisions than the groups that are actually affected by harm.

It should not be radical to suggest that individuals from minoritized groups are more qualified to speak about their oppression than members of privileged groups who either benefit from unjust power structures, or are actively involved in perpetuating that oppression. Of course, this does not mean that white writers should not discuss racism, or that male-identifying individuals should not speak about feminism, but rather that any editorial decisions about such work should prioritize writing about injustice by those who are actually affected by that injustice. Even if editors and publishers are not concerned with social justice, a basic consideration of quality would lead to the same conclusions about the importance of positionality.

And perhaps that is the most damning indictment of the critics of cancel culture: the system that they desire to preserve is one marked by inadequacy. Contrary to their claims of meritocracy, supposedly identity-blind traditional power structures perpetuated and continue to perpetuate cultures of mediocrity, where individuals from privileged groups were elevated above more deserving candidates from minoritized communities, often to prove the supposed intellectual diversity of such publications. For example, in The New York Times, the search for conservative writers led to Bret Stephens, who used his bully pulpit as a columnist for the paper of record to platform eugenics.

Of course, this does not mean that wrongful cancellations do not take place. The nature of society is that some voices will be always unfairly excluded, for editorial discretion is not always perfect. But even now, at almost all institutions, work that perpetuates harm is more common than work that is unfairly cancelled. And more importantly, the predominant “cancellation” in any society is that of marginalized groups. Thus, those questioning newer norms of accountability are not struggling for freedom of speech, but are attempting to preserve power structures that perpetuate injustice and the censorship that comes with it.