I went looking for what Turnitin says about code, expecting to find nothing. What I found was one sentence, and it is filed in a place you would never look for it.
The sentence is: "we are not pursuing ChatGPT code detection at this time."
It is not in the code section, because there isn't one. It is the last line of the answer to a completely different question — "Why is AI detection not being added to Gradescope?" The rest of that answer is about handwriting. Someone appended the code policy to it and moved on.
That is a strange place for it, and worth knowing, because Gradescope is where a lot of programming coursework gets handed in. If your implementation goes to an autograder and your written report goes to a Turnitin assignment, those are two pipelines doing two jobs, and the AI writing percentage you eventually see was computed on the second one only.
The other sentence, the one in the FAQ where you would expect it, says the model "does not reliably detect AI-generated text in the form of non-prose, or code, nor does it detect short-form/unconventional writing such as bullet points (short non-sentence structures)."
Read one line further, though. That is where it stops being trivia:
"This means that a document containing several different writing types would result in a disparity between the percentage and the highlights."
A coursework report is a document containing several different writing types by construction. So that disparity is not a bug you have hit. It is the normal state for this genre.
Here is what that means in practice, and it took me a while to see it properly.
Your report is two documents stapled together. One of them is listings, pseudocode, terminal transcripts, config, tables. The other is the prose between them: the design rationale, the algorithm walk-through, the complexity analysis, the evaluation discussion.
Only the second one is being measured. Turnitin defines it as "individual sentences contained in paragraphs that make up a longer piece of written work, such as an essay, a dissertation, or an article, etc." Your listings are not in the denominator. The FAQ says so directly: "This percentage is not necessarily the percentage of the entire submission."
So the number is a statement about a slice of your file, not your file. If your report is listing-heavy, that slice can be small. There is even a floor — a submission needs at least 300 words of prose in long-form format before a report is generated at all, which a page-heavy but prose-light report can sit under while looking perfectly substantial.
And now the part that actually bothered me.
The prose that is measured is the most uniform prose in the document. Turnitin publishes what its false positives tend to have in common: "content without a lot of structural variation, text that literally repeats itself, or text that has been paraphrased without developing new ideas."
Read that against a well-written technical document. Structural variation is the thing a reference section deliberately removes. Every function gets the same frame — name, parameters, return value, failure modes — so a reader who learns the frame once can skim the rest. Saying it the same way twice is how you tell a reader that two components behave the same way. A style guide telling you to vary the shape of each entry would be a bad style guide.
The algorithm walk-through has the same shape for a different reason: it restates in English what the listing above it already says, and restating something has nowhere to go stylistically. Every step opens with the step. Every sentence has the same subject. The paragraph is a transcript of control flow.
Two honest limits on that. Turnitin gives no frequency for those properties — only a list of what false positives "can include" — so nobody can tell you how much of the effect this accounts for. And you cannot reason backwards from the model, because the FAQ also says it "is not explicitly programmed to evaluate specific signals such as 'burstiness,' 'perplexity,' or other individual metrics sometimes referenced in public discussions."
But the reason I am writing this is the asymmetry, which gets much less attention than the percentage does.
The half of your report that is excluded from the measurement is not inert background. It is the half where one changed character makes the document wrong. And a revision pass — yours, a co-author's, a tool's — reads it whether or not anyone is scoring it.
getUser quietly becoming get_user means your prose now names a function your code does not define. A command with a flag reworded is a command that no longer runs. A log line inside quotation marks is a quotation, and rewording it makes it a fabrication. None of this is visible on a read-through, because it all reads perfectly well.
There is a subtler one. A technical document gives one concept one name. If a pass leaves hash map in section 3 and dictionary in section 4 for the same object, or alternates worker and thread, the document has stopped telling the reader whether those are one thing or two — and a marker cannot resolve it without reading your code. No score enters into that in either direction. It is just damage.
It cuts against a habit that serves you everywhere else. In most writing a defined term is a word choice. In this genre it is closer to an identifier.
So the checking pass I would actually run, in this order:
- Copy every command out of the document and run it in a scratch directory. Setup instructions rot first and get reread last.
- List the identifiers your code defines and grep the document for each one. Anything in the code and missing from the prose is a rename that happened without you.
- Pick your five load-bearing concept terms and search each. One name per concept, everywhere.
- Read every complexity claim against the function it describes.
O(n log n)and "efficient" are not interchangeable, and only one of them is markable. - Diff anything inside quotation marks against a real run. Log lines and error messages are quotations.
One last thing, and it is Turnitin's sentence rather than mine: the model "may not always be accurate (it may misidentify human-written, AI-generated, and AI-paraphrased text), so it should not be used as the sole basis for adverse actions against a student."
And separately from all of the above: what your department permits you to generate is set by its academic integrity policy, not by what a detector measures. Those are two different documents, and answering one does not answer the other.
Originally published at HumanPen: https://humanpen.net/blog/computer-science-report-ai-detection?utm_source=devto&utm_medium=article&utm_campaign=D206
Top comments (0)