why ai detectors* suck
most can be bypassed by simple tricks.
*In this blog, "AI detectors" refers to the tools or methods used to identify content created by AI, particularly in the form of text, such as ChatGPT-Zero. This is not to confused with AI models used to detect objects from pictures, like Midjourney.
Last week, my team and I turned in a thesis for our midterm project. After we turned that in, the results came back, showing that over 30% of the thesis was generated by AI. We got the member responsible to rewrite the entire thing.
According to a survey conducted by Intelligent.com in January 2023, 30% of college students in the US have used ChatGPT on written homework. Of this group, close to 60% use it on more than half of their assignments. Another survey by Forbes found that 43% of college students overall say they’ve had experience using AI tools like ChatGPT, and half of those acknowledge turning to those tools to work on assignments or exams.
When AI has this much popularity among students, it is no wonder that AI detectors like GPT-Zero or Turnitin are now essential tools for schools to know for sure whether the assignments of their students are done by machines or not.
But can they be really sure?
We thought they worked…
In order to understand how AI detectors work, we need to learn about the very thing they are trying to detect: generative AI models.
Generative AI models, like ChatGPT, or Midjourney, work by learning from a vast amount of data to create new content. It's similar to how an artist observes many paintings and then paints something unique based on what they've seen. Generative AI models analyze patterns, structures, and styles in the data they're trained on, allowing them to generate realistic text, images, or even music.
However, sometimes it's important to know if a piece of content was created by AI or a human. For example, in university, tests and assignments are designed to evaluate students’ performance and provide learning opportunities; however, they will become meaningless if the students’ don’t do those assignments themselves but use AI instead.
That's where AI detectors come in. AI detectors are designed to recognize the specific patterns and characteristics often found in AI-generated content. They compare the content in question with what they've learned from analyzing lots of AI-generated and human-created content.
An AI detector might notice certain repetitive phrases, an unusual choice of words, or a lack of logical inconsistencies that are common in human writing. By examining these clues, AI detectors can make an educated guess about whether the content was generated by AI or written by a human.
…but actually they kinda don’t
While AI detectors play a crucial role in identifying AI-generated content, it's important to acknowledge their limitations and the potential for bypassing them. Here’s why:
The ever-changing nature of AI:
AI detectors are designed using the patterns and traits found in already available AI-generated material. New variants and advancements do, however, appear as AI models and methodologies develop. It may be difficult for detectors to keep up with and correctly identify the most recent AI-generated material due to the versatility of AI models.
Think of this as being similar to a detective who has been taught to identify a criminal based on certain physical characteristics like height, hair color, and face structure.
The criminal may, however, vary their height, hair color, and even facial characteristics to continually change how they appear.
Because the criminal's look is continuously changing and the detective must continually modify their expertise to capture them, their work becomes very difficult.
Shapeshifters. Similar to this, it is challenging for AI detectors to keep up with the constantly shifting patterns and advancements in AI-generated material, making it challenging to precisely recognize the most recent variants created by developing AI models.
For example, this is how well GPT-Zero “detects” AI-generated material now:
This shows how hard it is to keep up with all the AI models.
Modification Techniques:
You can argue that GPt-Zero is now outdated, and soon individuals will be able to intentionally manipulate or "fool" AI detectors by using modification techniques. These techniques involve making subtle modifications to the AI-generated content that are not easily detectable to humans but can confuse the detectors.
This can be done either manually or, even better, with the help of some paraphrasers like Quillbot. Modification examples exploit vulnerabilities or limitations in the detector's algorithms, breaking the patterns that the models often make when creating content and therefore making it difficult for them to distinguish between AI-generated and human-generated content.
Creative Prompting:
AI models can sometimes generate content that mimics human writing very closely, if given the right prompt. If an AI-generated piece demonstrates sophisticated language use, context-awareness, and logical consistency, it becomes more challenging for detectors to confidently differentiate between AI and human content.
So, do we need them?
In my opinion, the advancement of technology, including AI, presents us with an opportunity to embrace it and foster innovation rather than view it as something to battle against.
Back to the survey of intelligent.com that I mentioned above, another interesting figure is that 3 in 4 ChatGPT users believe that using the tool can be considered cheating, but still use it anyway. Clearly, this is not a problem of awareness.
In the long term, trying to detect if student’’ works are created by AI or not is not a really plausible solution either. As previously stated, generative AI models are evolving faster and faster every week, and with modification techniques and creative prompting, I believe that AI detectors are not reliable fighters against those models.
Instead, I would suggest that we move away from focusing solely on ways to detect and counter AI-generated content and redirect our efforts towards designing more innovative challenges for students.
Some universities, like Flinders University, the University of Adelaide, and the University of South Australia, allow students to use ChatGPT as a writing prompter for assignments if they disclose it. The benefit of this approach is that you get a jumping-off point, but you still need to build your own argument and do the critical thinking.
University of South Africa senior lecturer in education futures, Vitomir Kovanovic, said all universities should allow AI and teach students how to use it. He stated that trying to stop AI is just a temporary solution.
“It’s like having a driving school but teaching people how to ride horses.”
Just take a look at the good old calculators; they eliminated the need for students to dedicate time to long division, consequently enabling teachers to assign more challenging tasks. In this age, trying to detect whether the students used calculators or not would be a very outdated approach. Who can say if, in the future, AI is not treated in the same way we treat calculators today?
In conclusion, my suggestion is that rather than perceiving AI as a threat to human creativity, we can view it as a tool that augments and enhances our capabilities. Schools can impose policies to limit the use of AI in the short term, but in the long run, everyone will eventually need to adapt.
Yeah, the ai detectors are a dead end. And I aslo have another proposal, which is to use AI transparently. For example, schools can allow the use of AI, but if someone uses it, they need to note below their essay how they used it. This will help make the usage process transparent and encourage responsible use.