AI Detector False Positives: Why Human Writing Gets Flagged

AI detectors don't prove who wrote something. They estimate how likely it is that a text was produced by an AI model, and that estimate is sometimes wrong. When a detector labels human writing as AI-generated, it's called a false positive. For a student, a job applicant or a freelance writer, one false positive can lead to a difficult conversation or worse.

This guide explains what the research shows, who gets flagged most often, and what to do on both sides of a flagged result.

Try it: use ProofAI's free ChatGPT checker, free for 10 checks a day with no login.

What the research shows

Three well-documented cases show why a score should never be treated as proof:

Also read: EU AI Act Article 50: AI Transparency Rules Explained

• Non-native English writers are flagged far more often. A Stanford study published in Patterns tested seven widely used detectors on 91 TOEFL essays written by non-native English speakers. On average, the detectors labelled 61.3% of those human-written essays as AI-generated. The same detectors classified 88 essays by US eighth-graders accurately. The authors concluded that "the design of many GPT detectors inherently discriminates against non-native authors" (Times Higher Education).
• Even the makers of AI models struggle. OpenAI's own AI text classifier correctly flagged only 26% of AI-written text and wrongly flagged 9% of human text. OpenAI withdrew it in July 2023 "due to its low rate of accuracy" (OpenAI).
• A small error rate adds up at scale. When Vanderbilt University disabled Turnitin's AI detector in 2023, it pointed out that a 1% false-positive rate applied to the 75,000 papers it submitted in 2022 would mean around 750 students wrongly flagged (Vanderbilt University).

Why human writing gets flagged

Most detectors look for text that is very predictable: common word choices, even sentence lengths, and few surprises. Some kinds of human writing naturally look like that:

• Writing in a second language. A smaller working vocabulary and safer sentence structures make text more predictable, which is the pattern the Stanford study found.
• Formal or template-style writing. Cover letters, lab reports, legal summaries and textbook-style answers follow set structures.
• Short texts. With only a few sentences, there isn't enough signal to judge, so scores swing widely. ProofAI asks for at least 50 words for this reason.
• Heavy use of grammar or paraphrasing tools. Tools that smooth out your writing also smooth out the variation detectors look for.
• Technical or subject-specific language. Fixed terminology and repeated phrases can read as "machine-like".

If your writing was flagged

1. Stay calm and ask what was flagged. Ask which passages were highlighted and what score was used. A single overall percentage tells you very little.
2. Show your process. Version history in Google Docs or Word, earlier drafts, notes, outlines and browser history are strong evidence of how you worked.
3. Offer to explain the work. Talking through your argument, sources and choices is often the most convincing evidence there is.
4. Point to the research. If you write in English as a second language, the Stanford findings above are directly relevant and worth sharing.

If you're reviewing a flagged result

1. Treat the score as a reason to look closer, not a verdict. Vanderbilt, OpenAI and the Stanford researchers all point the same way.
2. Read the highlighted passages yourself. Do they actually differ from the person's usual writing?
3. Compare with earlier, verified work by the same person.
4. Have a conversation before any decision. Ask about drafts and process.
5. Be extra careful with non-native English writers, short texts and formal formats, where false positives are most likely.
6. Write down your process. Use the same steps for everyone, so decisions are consistent and fair.

How ProofAI approaches this

ProofAI shows the passages that drove a score rather than only a single number, and our guidance is never to act on a score alone. If you're testing any detector, including ours, try it on a few pieces of writing you know are human first. That tells you more about its false-positive rate than any marketing claim.

Sources

• Liang, W. et al. (2023). GPT detectors are biased against non-native English writers. Patterns. Reported in Times Higher Education.
• OpenAI. New AI classifier for indicating AI-written text (with July 2023 update).
• Vanderbilt University. Guidance on AI detection and why we're disabling Turnitin's AI detector (August 2023).

Written and reviewed by the 4Born Content Writer Team. AI tools assisted with drafting; every statistic above links to its source.