AfriNR

Evaluating Native Reasoning in African Languages

Read the paper

A novel benchmark for evaluating AI systems on African culture and languages.

Enrique Molero1,+, Louis Powell1, Linda Petrini2, Audace Niyonkuru3, Samuel Rutunda3, Mugisha Boris Ishimwe3, Thomas Llewellyn4, Israel Abebe Azime5
1The GSM Association (GSMA); 2Independent Contributor; 3Digital Umuganda; 4SOAS University of London; 5Saarland University
+Main author. Correspondence: emolero@gsma.com

Key takeaways

Why this matters

Advanced AI models can default to Western-centric views, which hinders AI decolonisation.

Many African languages have limited data available for training AI. This makes it harder to build models that understand these languages and the cultural contexts in which people use them.

For AI to benefit the world, it must understand and respond to multiple languages and cultures around the world. Doing that involves native reasoning: understanding language and culture together to adapt and respond to local conventions.

What is AfriNR?

African Native Reasoning, or AfriNR, is a benchmark designed to measure this capability. It contains 1,082 questions written by local experts in nine African languages.

Language: Malagasy
Inona no mahazo raha mandaka rindrina?
What happens when you kick the wall?
Wrong!
Kicking a wall is “fady”, a taboo.
Astra answers what a kick does to a foot, but the question asks what Malagasy children are told happens.
Correct!
Ancestors told children this to keep the walls clean.

What do the questions test?

AfriNR contains four categories:

  • Grammar. Reasoning about a language’s rules and conventions.
  • Wordplay. Understanding riddles, double meanings and expressions.
  • Practices. Reasoning about local customs and traditions.
  • Maths. Solving problems involving local currencies, calendars or units of measurement.

How was AfriNR created?

We decided to take a fully human-centred approach, prioritising local people in writing, validating and solving the questions.

Native speakers recruited by Digital Umuganda write each question with its answer and explanation. Masakhane linguists and researchers, three per variety, keep, correct or remove it by majority vote, and corrected questions are re-reviewed until accepted. Native-speaker baseliners who never saw the questions answer them. Questions at least one baseliner answered correctly are kept, 1,082 questions in 9 languages; questions rejected at review or answered correctly by no baseliner are removed.

AfriNR

Removed

rejected by majority vote

Removed

no baseliner answered correctly

Dataset Generation

  • Native speakers
  • Recruited by Digital Umuganda
  • Write questions with answers and explanations

Validation

  • Masakhane linguists and researchers, 3 per variety
  • Keep, correct or remove by majority vote
  • Corrected and re-reviewed until accepted
  • Rejected questions removed

Solvability Check

  • Human Baseliners who never saw the questions answer them
  • Kept only if at least one answers correctly
  • Unsolved questions removed

Challenging and solvable

AfriNR contains questions that are challenging even for native speakers themselves. However, every retained question has been validated and answered correctly by at least one native speaker.

Results

Main results

We collected 5,270 responses from 49 native speakers to compare their performance with the eight frontier models evaluated in the paper.

Native speakers score 58.7% on average, compared with 44.7% for the models: 1.3 times as high. The strongest model in this comparison, GPT-6 Astra, scores 58.6%, approximately matching the average human performance. It remains below the best human score of 71.3%.

How we calculate the human scores

Accuracy measures how often an answer is correct. For Average Human Performance, we calculate each participant’s accuracy on the questions in their assigned set, then average these scores within each language variety. Best Human Performance uses the highest accuracy achieved by a participant for each assigned question set. These scores give each language variety equal weight, treating Swahili Tanzania and Swahili Mombasa independently.

The chart also includes specialists and their general-purpose counterparts evaluated across all nine languages. Choose a language or category to explore.

Closed-source models score 54.6% on average, while open-weight models, which developers can download and adapt, score 34.9%. This raises a further question: how much does the language of the question affect the answer?

Does asking in English help?

We presented the same 531 Practices and Maths questions in English and compared the answers with those given in the original African languages. The English prompts name the relevant community, so models still need to draw on local knowledge. We leave out Grammar and Wordplay because their challenge may not survive translation.

Accuracy rises for all eight models, with larger gains on average for open-weight models. This roughly halves the gap between the two groups on these questions.

Stronger linguistic performance is associated with a smaller linguistic gap. Models scoring higher on AfriNR’s linguistic questions tend to show a smaller accuracy gap between English and African languages on the same cultural questions (r = −0.85).

These gains suggest that part of the difficulty comes from understanding the original language. Higher English scores do not establish that models fully understand the culture.

African-language specialisation

Can further training in African languages reduce these gaps? We call this specialisation. We compare seven African-language specialists with the general-purpose models they were trained from, testing each pair in the languages the specialist supports.

Higher scores in African languages. Across all four categories, specialists score 11.7% on average, compared with 7.8% for their general-purpose counterparts: 1.5 times as high. The gains are concentrated in Grammar and Practices.

Lower scores in English. On the translated Practices and Maths questions, specialists score 25.5%, compared with 36.9% for their counterparts: 31% lower. All seven specialists lose accuracy in English, yet all seven still score higher in English than in the original African languages.

Explore ten selected questions below, switching between the original language and English to compare the two models’ answers and the judge’s verdicts.

A smaller language gap can hide a decline. Sunflower E2B’s English accuracy falls from 28.5% to 10.0%, while its African-language accuracy also falls, from 8.3% to 5.4%. Its scores become closer because performance deteriorates in both languages.

We cannot tell from these comparisons which part of the training caused the losses. To assess specialisation, we need to measure both the gains in African languages and any losses in English.

Conclusion

The higher accuracy of closed-source models may make them appear the best choice for deployment in Africa. However, limited research budgets, unequal access to compute and deployment costs can constrain adoption. Specialising open-weight models is a natural alternative.

Future work should test whether these trade-offs carry over to real tasks in African languages, where failures of native reasoning have practical and economic consequences. These evaluations should measure whether specialisation helps models communicate with African users, understand their needs and apply local knowledge to complete those tasks.

Read the paper for the full methodology and results.

References

  1. Having Beer after Prayer? Measuring Cultural Bias in Large Language ModelsTarek Naous, Michael J. Ryan, Alan Ritter and Wei Xu · ACL · 2024Finds a preference for Western cultural entities in multilingual and Arabic models, including in prompts grounded in Arab culture.
  2. Participatory Research for Low-resourced Machine Translation: A Case Study in African LanguagesWilhelmina Nekoto et al. · Findings of EMNLP · 2020Documents data and participation barriers in African language technology and develops machine translation through a participatory approach.
  3. MultiNRC: A Challenging and Native Multilingual Reasoning Evaluation Benchmark for LLMsAlexander R. Fabbri et al. · arXiv · 2025Introduces four categories of native reasoning in French, Spanish and Chinese. AfriNR adopts these categories for African languages and adds native-speaker baselines.
  4. AfriNR: Evaluating Native Reasoning in African LanguagesEnrique Molero et al. · Research paper · Data Collection · 2026Each retained question has at least one recorded correct native-speaker response. This does not mean every participant answered every question correctly.
  5. Sunflower: A New Approach to Expanding Coverage of African Languages in Large Language ModelsBenjamin Akera et al. · PMLR · AI for African Languages Conference · 2026Describes adapting Qwen models to improve comprehension across Ugandan languages. Provides background on the Sunflower family; the scores in this article come from AfriNR’s evaluation.
  6. AfriqueLLM: How Data Mixing and Model Architecture Impact Continued Pre-training for African LanguagesHao Yu et al. · ACL · 2026Studies further training on African-language data and how the training-data mix affects performance. Provides background on African-language specialisation; the scores in this article come from AfriNR’s evaluation.