AI vs Human Writing: Can You Really Tell the Difference?

Most people believe they can spot AI writing. The research says most people are wrong more often than they think, including experts. That matters, because a wrong guess can mean a student, a job applicant or a freelancer is accused of something they did not do.

This post sums up what published studies found, why the usual "tells" are unreliable, and how to check a piece of writing fairly.

Try it: use ProofAI's free ChatGPT checker, free for 10 checks a day with no login.

What the research says about human judgement

Several peer-reviewed studies have tested whether people can tell machine text from human text. The results are consistent: we are not good at it.

Non-experts guessed at chance. In a 2021 study presented at ACL, Clark et al. asked people to separate human writing from GPT-3 text across stories, news articles and recipes. Without training, accuracy was at "random chance level". Short training raised it to about 55%, and the gain was not significant across the three domains.
People rely on cues that mislead them. Jakesch, Hancock and Naaman ran six experiments with 4,600 participants, published in PNAS in 2023. Participants could not detect AI-written self-presentations in professional, hospitality and dating contexts. They treated first-person pronouns, contractions and mentions of family as signs of a human, which AI text can easily include.
Experts struggle too. Casal and Kessler asked 72 reviewers from applied linguistics journals to judge research abstracts. They identified AI versus human writing correctly 38.9% of the time overall (University of Memphis summary).
Specialists do better, but still make errors both ways. In Gao et al. (2023, npj Digital Medicine), blinded human reviewers correctly identified 68% of ChatGPT-generated medical abstracts, and wrongly labelled 14% of real abstracts as generated.

Note what that last figure means in practice.

Why the classic "AI tells" are unreliable

Older advice lists signs such as flawless grammar, formal tone, repeated sentence patterns and a lack of personal stories. These can point to AI. They also describe a lot of careful human writing.

Want to check your own content? Try ProofAI free 10 checks a day, no login needed.

Polished grammar is common in edited, professional or technical text.
Formal, predictable phrasing is how many people are taught to write, especially in a second language.
No personal anecdotes is normal in reports, abstracts and product copy.
Personal touches do not prove a human wrote something. The Jakesch study found people were fooled when AI text used exactly these features.

Treat these patterns as reasons to look closer, not as conclusions.

Can AI detectors do better than people?

Detection software can help, but it has well-documented limits.

Also read: AI Detection for Universities: A Practical Policy Guide

OpenAI launched its own AI text classifier in January 2023. On its test set it correctly flagged 26% of AI-written text and mislabelled human text as AI 9% of the time. OpenAI warned it was "very unreliable" on texts under 1,000 characters, and withdrew it on 20 July 2023 "due to its low rate of accuracy" (OpenAI).

Detectors can also be biased. A Stanford study in Patterns by Liang et al. tested seven widely used detectors on 91 TOEFL essays written by Chinese students. The average false-positive rate was 61.3%, while essays by US eighth-graders were classified correctly (Times Higher Education). In the paper, the authors found that the essays all seven detectors flagged had significantly lower text perplexity, and concluded that detectors "may penalize non-native writers with limited linguistic expressions" (Liang et al., arXiv). In plain terms, more predictable wording reads as machine-like to a detector. The same study found that asking ChatGPT to use more literary language pushed detection rates close to zero.

So a detector score is a probability estimate from a model with its own blind spots. It is not proof.

A fair way to check a piece of writing

If you need to judge whether a document was AI-written, use several kinds of evidence and keep a human in charge of the decision.

Decide why it matters. Is AI use banned here, allowed with disclosure, or fine? Many uses, such as grammar help, are acceptable. Judge against the actual rule.
Check the facts and references. Fabricated sources, wrong dates and confident errors are concrete and verifiable. They are stronger evidence than tone.
Compare with known work by the same person. A sudden change in voice, vocabulary or skill level is worth asking about. Make sure the comparison samples are similar in type and length.
Use a detector as one signal. Prefer tools that show which passages drove the score so you can read them yourself. Run longer samples where possible, since short texts are less reliable.
Talk to the author. Ask them to explain their argument, their sources or how they wrote it. Drafts, notes and version history are good evidence of authorship.
Never act on a score alone. Record what you checked and why, and give the person a chance to respond.

If you use ProofAI's text detector for step 4, it shows the passages behind the score so you can review them, and its results are probabilities, not verdicts.

What still marks good human writing

Even if you cannot reliably tell who wrote a text, you can judge whether it is useful. Strong writing tends to contain things that are hard to fake without real knowledge: first-hand detail you can verify, specific examples, accurate sources, and a clear point of view the author can defend when asked.

For writers, that is also the best protection against false flags. Keep your drafts and notes, cite your sources, and write from what you actually know.

Sources

Clark, E. et al. (2021). All That's 'Human' Is Not Gold: Evaluating Human Evaluation of Generated Text. ACL 2021. https://aclanthology.org/2021.acl-long.565
Jakesch, M., Hancock, J. and Naaman, M. (2023). Human heuristics for AI-generated language are flawed. PNAS. https://arxiv.org/abs/2206.07271v4
Casal, J. E. and Kessler, M. (2023). Can linguists distinguish between ChatGPT/AI and human writing? Research Methods in Applied Linguistics. https://digitalcommons.memphis.edu/facpub2/10
University of Memphis press release (7 Sept 2023). https://www.memphis.edu/mediaroom/releases/2023/september/ai-study-by-uofm-dr-j-elliot-casal.php
Gao, C. A. et al. (2023). Comparing scientific abstracts generated by ChatGPT to real abstracts with detectors and blinded human reviewers. npj Digital Medicine. https://pmc.ncbi.nlm.nih.gov/articles/PMC10133283
OpenAI. New AI classifier for indicating AI-written text (updated July 2023). https://openai.com/index/new-ai-classifier-for-indicating-ai-written-text/
Liang, W. et al. (2023). GPT detectors are biased against non-native English writers. Patterns. https://arxiv.org/abs/2304.02819
Times Higher Education (11 July 2023). AI text detectors 'biased against non-native English speakers'. https://www.timeshighereducation.com/node/727071

Written and reviewed by the 4Born Content Writer Team. AI tools assisted with drafting; every statistic above links to its source.