Rethinking AI Scale Through Content Moderation

Discussions of AI expansion often adopt an anticipatory perspective, framing present concerns as temporary costs of inevitable progress. But using the example of content moderation, this narrative risks normalising exploitation in the present.

Early open-source and commercial platforms were largely reliant on human workers to manually categorise and label vast amounts of graphic, explicit material in order for the AI to recognise and filter it. Sociologist Milagros Miceli describes the work model as “comparable to any other lethal industry,” citing the low pay, margins, long hours, and psychological harm that accompanies it.

But as machine-learning systems improved, the vast majority of content moderation and AI training has been handed over to automated models instead of humans. This transition frames harmful moderation labour as a temporary cost that will eventually be eliminated through scale, supporting the techno-optimistic narrative. However, this interpretation can be misleading, as it ignores the nuances of how a large part of human labour is transformed and redistributed.

First, it’s not necessary to develop a type of AGI or model with perfect contextual understanding to create a system of full automation. In practice, platforms operate on acceptable error rates rather than perfection. But even highly accurate systems continue to struggle with edge cases of political speech, satire, or rapidly evolving extremist content. As automated systems can handle the bulk of the content, human moderators still increasingly review the most ambiguous and often psychologically taxing material. In this case, automation concentrates the most harmful forms of labour at higher margins.

Moreover, this redistribution of labour also redistributes responsibility. Since moderation is a normative process, what constitutes“extremism,” “hate speech,” or “harm,” varies across jurisdictions over time. When such classification decisions are delegated to automated systems, the problem of accountability arises. Since the decision is the result of a design involving many actors, accountability is not naturally obvious. Determining responsibility depends on the evaluator’s normative framework, moral and political assumptions.

These issues are not speculative future concerns, they already exist in the present day, yet rarely alter the dogmatic scale approaches where the end justifies all means. One way to understand this disconnect is to examine the institutional choices that produce these harms. As Karen Hal notes in her book Empire of AI, the social costs of AI are not inherent but deliberate, structural choices. She uses a forest analogy to illustrate this, where, to reach the other side, or reap AI’s benefits, one can either bulldoze or carefully wind through the forest. One method is significantly easier, but disastrous. In contrast, carefully winding would resemble the approach of research communities like the DAIR Institute, which promotes the development of smaller, more specialised models. By prioritising efficiency over scale, they allow for greater control and effectiveness, and can be deployed and regulated more ethically within local contexts. Yet, this solution seems slightly unconvincing as unrealistic for many countries in the current global context, where intense economic and geopolitical pressures reward speed and dominance.

The issue then is not if AI can eliminate human moderation entirely, but whether we are too quick to excuse its social costs under the near-theological assumption that scale will ultimately resolve them. Although public awareness and calls for regulation are increasing, so long as expansion is treated as self-justifying, its social costs will continue to be absorbed.

This article was first published as part of the Sundial Press printed edition in Spring 2026