Chatbots: (S)elected Moderation

Measuring how Copilot, ChatGPT, and Gemini moderate election-related content across different languages and electoral contexts

31-07-2024

Project overview

Report cover

After our investigation revealed Microsoft Copilot's 30% error rate on election questions and our work with Nieuwsuur exposed how chatbots can support disinformation campaigns, major AI companies implemented election content moderation. This report evaluates the latter across different scenarios.

Key findings reveal:

  • Effectiveness varies dramatically between chatbots: Gemini achieved 98% moderation consistency, Copilot only reached 50%, while OpenAI's web version of ChatGPT showed no additional election-related moderation
  • Language disparities are significant: For EU election prompts on Copilot, moderation was highest for English (90%), followed by Polish (80%), Italian (74%), and French (72%)
  • Moderation falls below 30% for Romanian, Swedish, Greek, Dutch, and even German (28%) despite being the EU's second most spoken language
  • For identical prompts in the same language regarding EU versus US elections, moderation rates vary substantially
  • Implementation inconsistencies exist between web and API versions: Gemini's web safeguards are absent in its API version.
AI Forensics is proud to be trusted by