Over 10 weeks from mid-July, our team of fact checkers and data scientists asked five widely used AI models the same 22 questions every day: Gemini 3.1, ChatGPT, Grok, DeepSeek and Claude.
We scored tens of thousands of individual claims against five criteria: factuality, transparency, timeliness, consistency and civic responsibility.
No model repeated the core piece of misinformation it was tested on, but the exercise exposed several vulnerabilities and points of failure.
Among the findings:
▪️ Rules not yet in force were reported as law. Chatbots said bank cards could be used as voter ID, and that a £10,000 asylum rule was in force, despite it not having been adopted.
▪️ Outdated figures were presented as current. Several models quoted a weekly asylum support rate of £8.86 after it had risen to £9.95.
▪️ UK-wide answers ignored differences between the nations. Voter ID rules vary across the UK, but models frequently omitted the caveat needed to make their answers accurate.
▪️ The same question produced different answers on different days. One model correctly identified 2025 as the answer to when England last won a major football trophy, but gave 2022 on other days.
So, we’re calling on AI companies to ensure models check live, authoritative sources for time-sensitive facts; distinguish clearly between rules in force and proposals; retain essential caveats even when this makes the answers longer; and be able to account for the sources they cite.
This is the first in a series of reports we plan to publish, and we will have much more to say in the months ahead. This project has been made possible thanks to a grant from the The Patrick J. McGovern Foundation.
Read more about our methodology and key findings here: https://lnkd.in/e5SjGcbB