How Accurate Are AI Detectors? What the Evidence Says

The honest answer is: accurate enough to be a useful signal, not accurate enough to be proof. That gap matters most when a score is used to make decisions about a person, such as a student, a job applicant or a freelancer.

This post looks at what has actually been published about detector accuracy, why results vary, and how to use a score without misusing it.

Try it: use ProofAI's free AI checker, free for 10 checks a day with no login.

Two kinds of error

Every detector makes two kinds of mistake:

• False positive: human writing labelled as AI. This is the error that harms people.
• False negative: AI writing labelled as human. This is the error that lets content through.

Vendors tune detectors to trade one against the other. Turnitin says openly that it aims to keep false positives low and that, as a tradeoff, some AI-written text may go undetected (Turnitin FAQ). Any accuracy figure that does not separate the two errors tells you very little.

What published numbers show

OpenAI's classifier. In January 2023 OpenAI released a tool to flag AI-written text. On its English challenge set, it correctly identified 26% of AI-written text, and wrongly labelled human text as AI 9% of the time. OpenAI withdrew it on 20 July 2023 "due to its low rate of accuracy" (OpenAI). If the company that built ChatGPT could not make a reliable detector at the time, claims of near-perfect accuracy deserve scepticism.

Turnitin's figures. In May 2023 Turnitin reported a document-level false positive rate of less than 1% for documents with over 20% AI writing, and a sentence-level false positive rate of about 4%. In the same update it raised the minimum length for analysis from 150 to 300 words and added an asterisk to scores under 20% to signal lower reliability (Turnitin).

What a small rate means at scale. Vanderbilt University applied Turnitin's 1% figure to the 75,000 papers it submitted in 2022 and estimated that around 750 could have been wrongly flagged. It disabled the AI detector in August 2023 (Vanderbilt). The University of Pittsburgh's Teaching Center did the same, citing a "substantial risk of false positives" (University Times, Pitt).

Also read: Signs Content Was Written by ChatGPT (and Their Limits)

Bias against non-native English writers

The most cited independent study is by Liang and colleagues at Stanford, published in Patterns. Seven widely used detectors were tested on 91 TOEFL essays by Chinese students. The average false positive rate was 61.3%. The same detectors classified 88 essays by US eighth-graders accurately. When the researchers prompted ChatGPT to use more literary language, detection rates fell to near zero (Times Higher Education).

That one study shows both failure modes: human writing flagged because it is plain, and AI writing missed because it was prompted to sound different.

Why results vary so much

Text length. Short text gives a detector less to work with. OpenAI called its classifier "very unreliable" below 1,000 characters, and Turnitin needs 300 words of long-form prose for an English report.

Language. OpenAI recommended using its classifier only for English. Turnitin lists three supported languages: long-form English, Spanish and Japanese (Turnitin FAQ). Outside the languages a tool was built for, treat results with extra caution.

Editing. OpenAI noted that AI-written text "can be edited to evade the classifier." Mixed documents, where a person edits a generated draft or uses AI to polish their own writing, are the hardest cases for any tool.

Writing style. Formulaic genres such as lab reports, legal text and technical documentation are predictable by nature, which is close to what detectors look for.

Model updates. Generators change often. A detector tested against older models may perform differently on newer ones, so ask when a vendor's accuracy figures were measured.

How to read a detector score

1. Treat it as a probability. "85% likely AI" is a model's estimate, not a finding that 85% of the text was generated.
2. Check the conditions. Is the text long enough? Is it in a supported language? Is it mostly prose?
3. Look at which passages were flagged. A tool that shows its reasoning lets you judge whether the flagged sentences are actually suspicious.
4. Look for independent evidence. Fabricated citations, factual errors, or a sharp change from someone's earlier writing.
5. Never act on the score alone. Turnitin's own guidance says the score "should not be used as the sole basis for action."

Questions to ask any vendor

• What are your false positive and false negative rates, measured separately?
• On what data, in which languages, at what text lengths, and when?
• Has anyone outside the company tested the tool?
• How do you label results: as probabilities, or as verdicts?

If a vendor offers one headline accuracy number with no detail behind it, that tells you something.

Where we stand

ProofAI is a detector too, and the same limits apply to it: results are probabilities, not proof. The main text detector needs at least 50 words and highlights the passages that drove the score, so you can review the reasoning yourself. ProofAI sends the text or files you check to AI model providers to analyse them and keeps a record of each check (date, type and score, never your content); see our privacy policy.

Sources

• Turnitin, "AI writing detection capabilities FAQs": https://guides.turnitin.com/hc/en-us/articles/28477544839821-Turnitin-s-AI-writing-detection-capabilities-FAQs
• OpenAI, "New AI classifier for indicating AI-written text" (with July 2023 update): https://openai.com/index/new-ai-classifier-for-indicating-ai-written-text/
• Turnitin, "AI writing detection update from Turnitin's Chief Product Officer" (23 May 2023): https://www.turnitin.co.uk/blog/ai-writing-detection-update-from-turnitins-chief-product-officer
• Vanderbilt University, "Guidance on AI detection and why we're disabling Turnitin's AI detector" (16 August 2023): https://www.vanderbilt.edu/brightspace/2023/08/16/guidance-on-ai-detection-and-why-were-disabling-turnitins-ai-detector/
• University Times, University of Pittsburgh (8 September 2023): https://utimes.pitt.edu/news/gen-ai-detection-software
• Times Higher Education on Liang et al., Patterns (11 July 2023): https://www.timeshighereducation.com/node/727071

Written and reviewed by the 4Born Content Writer Team. AI tools assisted with drafting; every statistic above links to its source.