I Built Turnitin0 to Study Turnitin AI Detection — Research, Humanization, and an Open-Source Agent Skill

James Peng — Fri, 21 Aug 2026 07:17:45 +0000

I originally built Turnitin0 as a third-party Turnitin AI and similarity checking service, but the project gradually became a much more interesting technical experiment around AI-generated text detection. What started as a simple question — "Can I check a document before submitting it?" — turned into a larger investigation into how AI detection works, what makes AI-generated writing statistically different from human writing, and how those signals change when an LLM modifies existing text.

You can try the Turnitin AI and plagiarism check if you want to use the product, but this post is mainly about the technical side of the project: the signals behind AI detection, the experiments I've been running, how the humanizer works conceptually, and how I turned the workflow into an open-source skill for coding agents.

1. How AI Text Detection Works

A language model generates text by repeatedly predicting the next token. Given a sequence of previous tokens, the model assigns probabilities to possible next tokens and then selects or samples one of them. This process naturally produces statistical patterns in the resulting text. One of the most commonly discussed signals is perplexity, which roughly describes how predictable a sequence of tokens is to a language model. Text containing highly predictable token sequences tends to have lower perplexity, while text containing more surprising word choices tends to have higher perplexity.

Perplexity alone, however, is not enough to determine whether text was written by an AI. Academic writing can naturally have low perplexity because it uses conventional terminology and predictable sentence structures. On the other hand, an LLM can deliberately produce more varied vocabulary and sentence structures. This is where other characteristics become relevant, including burstiness, sentence-length distribution, vocabulary variation, syntactic patterns, repetition, and consistency across paragraphs.

At a larger scale, AI-generated documents can also exhibit relatively consistent rhetorical and structural patterns. Transitions may follow predictable templates, sentences may have similar grammatical structures, and paragraphs may maintain unusually consistent levels of complexity. Human writing is often more irregular because it is produced through an iterative process involving revisions, changes of direction, personal writing habits, and occasionally imperfect or unexpected phrasing.

So I don't think AI detection is best understood as a simple keyword or phrase matching problem. A useful detector needs to combine signals at multiple levels, from token probabilities and sentence statistics to paragraph-level and document-level characteristics. More importantly, these signals need to be evaluated probabilistically rather than treated as absolute proof that a particular person used AI.

2. Building a Research Dataset

Once I started testing Turnitin AI detection, I realized that individual examples weren't particularly useful. If one essay receives a particular AI percentage, that doesn't tell us much about the underlying detection system. To make the experiments more meaningful, I started building datasets containing different categories of writing and testing them under controlled conditions.

The Turnitin0 Research Center contains experiments comparing AI-generated and human-written documents. I've tested essays generated by models including GPT-5.6-Sol, Claude Fable-5, and Gemini, and compared them with human-written datasets such as PLOS and undergraduate writing collections. The purpose is not simply to find which model "beats" a detector, but to understand how different types of text behave.

One particularly interesting category is AI-polished human writing. Instead of asking an LLM to generate an essay from scratch, you can start with a genuinely human-written document and ask the model to improve its grammar, structure, vocabulary, or clarity. The resulting document is neither purely human-written nor purely AI-generated. It represents a much more realistic form of AI-assisted writing.

In my experiments, this distinction matters. Completely AI-generated documents can produce much stronger and more consistent detection signals, while AI-polished human writing can produce a mixed result where some sections are detected and others remain classified as human. This suggests that looking only at the final document-level percentage can hide important information about what is actually happening inside the document.

That's one reason I'm publishing the experiments and datasets rather than only reporting a single "accuracy" number. I'm interested in how the classification changes when the underlying text generation process changes.

3. Humanization as a Text Transformation Problem

The humanizer side of Turnitin0 came directly from this research. If AI-generated text contains measurable statistical characteristics, then rewriting the text changes those characteristics. The interesting technical problem is deciding what should be changed and how much it should be changed without destroying the original meaning.

A simplistic approach would be synonym replacement, but that doesn't address the deeper structural characteristics of generated writing. A more sophisticated transformation can modify sentence boundaries, syntactic structures, transitions, vocabulary selection, paragraph organization, and sentence-length distribution. These changes can make the resulting text statistically different from the original generation while keeping the underlying ideas relatively stable.

This can be viewed as a constrained optimization problem. On one side, there is semantic preservation: the rewritten document should still communicate the same ideas. On the other side, there is stylistic transformation: the statistical characteristics associated with the original AI generation need to change sufficiently. If the transformation is too aggressive, the meaning and quality of the document can deteriorate. If it is too conservative, the resulting text may retain too many characteristics of the original generation.

This is also why I don't treat a humanizer as a one-shot text rewriting function. The workflow can be iterative: transform the text, run a detection check, inspect the result, and revise again when necessary. The detection result becomes feedback rather than simply being the final output.

4. Turning the Workflow Into an Open-Source Agent Skill

The next step was making the entire workflow accessible to AI coding agents. I created an open-source Turnitin0 skill on GitHub that currently supports Codex, Claude Code, and Cursor.

The architecture is intentionally straightforward. The main skill implementation lives under skills/turnitin0/, with Python scripts handling the individual operations. The workflow is divided into components for authentication, Turnitin checking, humanization, payment, and downloading the resulting files. Different agent environments use their corresponding skill configuration and entry points, while the underlying Python implementation provides the actual functionality.

This means an agent can receive a request such as "check this PDF with Turnitin0" and invoke the appropriate workflow rather than requiring the user to manually open a website, upload the document, wait for the result, and download the report. The same approach can be used for humanization and retrieving the resulting document.

The repository is MIT licensed, and installation is deliberately simple:

npx skills add pengchenghaoche51/turnitin0_skill

I think this direction is interesting because AI agents are increasingly becoming the interface through which people interact with software. Instead of building a separate API integration for every workflow, an agent skill can expose a set of capabilities that the agent can discover and invoke when needed. Turnitin0 is a relatively small example of this idea applied to document analysis and transformation.

What's Next

Turnitin0 has gradually evolved from a simple checking service into three connected projects: a Turnitin AI and similarity checking workflow, a research project investigating AI detection behavior, and an open-source interface for AI coding agents.

The research is probably the part I'm most interested in continuing. As language models become better at producing varied and natural writing, the statistical differences between human and AI-generated text may become increasingly subtle. At the same time, AI-assisted writing introduces an even more difficult category where a human document can be substantially modified by an LLM without becoming completely AI-generated.

The interesting question is therefore not simply "Can an AI detector detect this text?" but rather: what measurable properties caused the detector to make that decision, how stable are those properties under different transformations, and how do they change as language models evolve?

That's the direction I'm taking Turnitin0 in: part SaaS tool, part research project, and increasingly part open-source agent infrastructure.