Moderation Built by Language Communities: The Models Are Feasible, the Handover Is Unproven

The problem

An independent audit published on 8 September 2026 found that the most widely used generic hate-speech classifier recovers only about a tenth of hate speech in English translations of Amharic and Afan Oromo posts, and that the dominant error across every tool tested was under-detection (independent audit, abstract)¹. The audit tested three classifiers rather than a named platform's system, so it shows a risk, not a platform error rate.

Starting point. The anchor paper found that automated moderation is in wide use on large platforms, that most accuracy figures are reported by the platforms themselves, that no independent platform-level audit of machine-learning accuracy was found, and that smaller languages are served worst². Meta stated in March 2026 that its newer systems work in languages spoken by 98% of people online, against about 80 languages before (operator-reported; test design not described)³. This paper asks a different question: who builds and audits the models for a smaller language, and what route could carry their work into a platform's decisions.

Why data is the bottleneck. A December 2024 report from the Center for Democracy & Technology on Kiswahili found that scarce public text hinders the development of machine-learning models, which "continue to be optimized primarily for high-resource languages like English" (independent research report)⁴. In a survey of 143 frequent social media users in Kenya and Tanzania, run from July to September 2024, about 70% had reported content at least once and about 46% of those said action was taken (self-reported survey)⁴. Human cover is also uneven. Self-reported figures for April to September 2023 show TikTok with 2,137 English-speaking reviewers and 69 for Bulgarian, and no platform reported the language of the content it moderated in the EU database⁵.

In short, people who write in smaller languages are served by systems trained mostly on other languages, and the public record is too thin to say how large the gap is.

What already works

Four building blocks exist today. None has been shown to be taken up by a platform.

Community-labelled datasets. AfriHate is a collection of hate-speech and abusive-language datasets in 15 African languages, built from posts on X collected between 2012 and 2023, each labelled by native speakers, with a native-speaker language lead per language to control quality⁶. The number of annotators ranged from 3 to 11 per language, three or four labelled each post, and chance-corrected agreement between labellers ranged from 0.46 (Somali and Tigrinya) to 0.81 (Kinyarwanda and isiZulu) (study, self-measured)⁶. Summing the per-language counts in the paper's table gives about 90,000 labelled posts (computed from the table)⁶. The lower agreement means that, for some languages, labellers disagreed more often about what counts as hate, while the overall result shows that native-speaker labelling across many languages is possible.

Experience-led annotation. Uli is a project of the Indian civic-technology organisation Tattle, with some 20 co-authors from universities, women's organisations and media groups⁷. Its dataset covers Hindi, Tamil and Indian English and was drawn from close to 1.3 million tweets from 2018 to 2021; gender and sexual rights activists and researchers annotated it between March and July 2022 and were paid in proportion to the posts they labelled, and the budget set the size at roughly 8,000 posts per language (study)⁷. The data is released under a CC BY 4.0 licence with annotator-level labels⁷. The authors report that Tamil had the highest model performance and English remained the hardest⁷. The tool built on it is an open-source browser extension that lets users remove derogatory slurs, archive abusive posts and coordinate reporting within their networks (organisation description)⁸.

Human-led local moderation. The Kiswahili report describes JamiiForums, a Tanzanian platform with approximately three million users per day, which uses automation only to remove spam, insults and content reported many times, with a moderator making the final decision⁴. The report calls the approach representative and fair but questions whether platforms with much larger user bases could copy it⁴. It is a reminder that a largely non-AI option already works at one scale.

Open safety infrastructure. ROOST, a nonprofit launched at the Paris AI Action Summit, reported in December 2025 that it had introduced Osprey (an investigation tool donated by Discord), Coop (a review platform it acquired), gpt-oss-safeguard (a "bring your own policy" reasoning model released with OpenAI) and a community to "co-create, share, and refine open safety AI models" (organisation-reported)⁹. The page does not mention language coverage⁹. Separately, the European Commission's delegated act on data access for vetted researchers under Article 40 of the Digital Services Act entered into force on 29 October 2025 and sets up a portal through which researchers request data from very large platforms and search engines (regulator news item)¹⁰.

What does not exist, as far as the sources show, is a documented case of a platform putting a community-built dataset or model into its own moderation decisions.

The possibility

One way this could work is a three-party arrangement for a single language. A university department, a newsroom and a civil-society group could agree a labelling guide, pay native-speaker annotators, and publish a dataset and a classifier under an open licence, on the AfriHate and Uli pattern. An independent group could then test that classifier, and the platform's own system, against a held-out sample, as the Ethiopian audit did for generic tools. A platform could then use the result in one of three ways, in rising order of commitment: as a test set against which it reports its own accuracy in that language; as an extra signal that sends borderline posts in that language to a human reviewer; or as a component of its production classifier.

The first form may be the most realistic starting point, because it asks the platform to publish a number rather than hand over a decision. If it held, the community's main product would be a trusted yardstick, with a deployable model as a later step.

The role of AI. The AI part is the machine-learning classifier trained on the community labels. Almost everything else is ordinary work: agreeing categories, recruiting, paying and protecting annotators, licensing the data, building review queues and appeals, signing contracts and finding money. If the Kiswahili report is right that scarce data and scarce local expertise are the bottleneck, the model would be the smaller part of the effort.

What would have to be true

1. Labelled data can be produced in the language at acceptable agreement between labellers. AfriHate reached 0.46 in two languages and 0.81 in two others, so the outcome would depend on the language and on how clearly the categories are defined⁶.
2. Annotators are paid fairly and protected, and the funding continues past a single project. Uli's paper treats compensation as an open question⁷, and Kiswahili researchers described turning to students, friends and in some cases AI tools for annotation because it is costly⁴.
3. Models are tested independently in each language. The Ethiopian audit found that locally oriented classifiers did far better on Amharic but collapsed on Afan Oromo, so success in one language did not carry over to the next¹.
4. A platform is willing to take an outside signal, and a feedback channel works. This is a condition the evidence casts doubt on (see the next section).
5. Audit groups can obtain platform data to test against. The Article 40 route is limited to vetted researchers and very large platforms, and the source read does not say whether language-community groups would qualify¹⁰.
6. Dialects, code-mixing and non-text content are handled. Kiswahili researchers said the fast-changing street language Sheng complicates data collection⁴, and the Uli authors note that their annotation covers text only although much abuse is in images and video⁷.
7. Data licences let a platform reuse the work, and platform use does not turn unpaid community labour into a free input.

What could stop it

What the evidence shows

No evidence was found in the sources read that any platform has adopted a community-built moderation model or dataset. The Uli sources describe a tool for users, not a platform integration⁷⁸.

The existing outside channel is slow and uneven. An Internews study of Meta's Trusted Partner Programme, which lets local civil-society organisations escalate reports, found 465 partner organisations in 122 countries as of February 2023 and response times that varied from a single day to as long as eight months for ostensibly similar reports (independent report; interviews and surveys with 24 partners; data over three years old)¹¹. Ukrainian partners could expect a response within 72 hours, while reports on the Tigray war in Ethiopia could go unanswered for several months¹¹. Meta said it generally expects reports to be reviewed within one to five days and acknowledged past problems (operator-stated, February 2023), and it declined to give response-rate data¹¹. The report does not say whether this has changed since.

Transfer between languages is weak. The Ethiopian audit found that the dominant error across every tool was under-detection, and that a locally oriented tool collapsed on one of the two languages¹. The 98% language-coverage claim from Meta and the audit are not like-for-like, because the audited tools are not Meta's systems¹³.

Labelling is hard and costly. Agreement between labellers fell to 0.46 in two AfriHate languages⁶. The Uli authors state that creating the dataset was labour-intensive and that approaches beyond majority voting are needed to handle disagreement⁷.

Open safety tooling is not yet language-specific. The ROOST review does not mention languages⁹.

The plain reading of these facts is that data and labelling methods exist, while the evidence for a working path into platforms is missing or points the other way.

Reasoning

The challenge question is what would make this less likely, less useful or more harmful than it first appears. Five points stand out.

First, who pays. If a platform with large revenues takes a community dataset without funding its upkeep, the arrangement could turn local volunteers and under-funded universities into an unpaid supply chain. The condition on fair pay would then fail.

Second, who bears the risk. Annotators read abusive material, and the Kiswahili report records limited wellbeing support for paid moderators⁴. Community groups might carry the same exposure with fewer protections. A model trained on local labels could also be turned against the community, for example by a state or a platform using it to over-remove political speech in a language few reviewers can check.

Third, incentives. A platform's interest is in cost and legal exposure. If its own multilingual systems keep improving, as Meta says they do³, a community model might be judged redundant. The better role for the community might then be as independent auditor, not supplier.

Fourth, simpler options. Hiring and training more native-speaking reviewers, and fixing the trusted-partner channel so that reports from Ethiopia are answered as fast as those from Ukraine, would need no new model. If response times were the real bottleneck, a new classifier would not remove it.

Fifth, scale. JamiiForums works with a human-led approach at its size⁴, and the same approach might not carry to platforms with hundreds of millions of users in dozens of languages.

Signs to watch

• A platform publishing language-level moderation accuracy, or stating that it uses an external community dataset or model.
• The first requests by vetted researchers under the Article 40 delegated act that concern moderation in smaller languages¹⁰.
• ROOST Model Community releases that list the languages they cover⁹.
• Independent follow-ups to the Ethiopian audit that extend it to more languages¹.
• Newer public data on trusted-partner response times, since the figures here date from February 2023¹¹.