In collaboration with Algorithm Watch, we tested Microsoft’s Copilot (Bing Chat), a large language model chatbot, during the 2023 Swiss Federal Elections and German state elections in Hesse and Bavaria. Over two months, we prompted the chatbot with over 1,000 election-related queries on dates, candidates, polling, and controversies. Local experts analyzed responses for errors and reliability.

Our findings:

• One-third of Bing Chat’s answers contained factual errors, including incorrect election dates, outdated candidates, or fabricated controversies.
• The chatbot evaded questions 40% of the time. Many evasive answers were about simple election details, undermining trust in the tool as a reliable source.
• The chatbot’s factual errors also pose reputational risks, as it often attributed false information to credible sources. In some cases, it even fabricated scandals about candidates.

As people increasingly turn to LLM-powered search engines for information, it is more important than ever to ensure safeguards are in place to mitigate its risks during elections and beyond.