Ever since the launch of ChatGPT and its rivals, opinions on the AI chatbot have been divided. While some feel it is vital and time-saving, a few others are concerned with people using AI to spread misinformation and plagiarize work. OpenAI, the company behind the viral AI bot ChatGPT and text-to-image generator DALL-E, has released a new AI Classifier tool to identify AI-written text. In an official press release, OpenAI details how the Classifier was trained, its limitations, and other metrics. Check out all the details below.
OpenAI’s Classifier Distinguishes AI Text from Human Writing
In a recent press release, the company detailed how this new AI Classifier tool is meant to help people in identifying AI text. The tool does that by receiving input from users and then running it through its trained data to classify it as AI or human-generated content.
The AI Classifier is a language model and has been trained on a dataset that comprises pairs of human and AI-written text on the same topic. OpenAI notes that human data has been collected from various sources that they believe to be written by humans. This text has been divided into prompts and responses upon which the AI text was generated to gather a full dataset.
The AI Classifier Isn’t Fully Reliable Yet
While the new tool is meant to ease our lives (especially teachers and professors) when it comes to AI, the company has clearly stated that the AI Classifier isn’t fully reliable and prone to errors. In the company’s internal evaluations, the classier correctly identified 26% of AI-written text as “likely AI-written”.
It further incorrectly identified 9% of the human text as AI-written. While the margin of error is fairly low, it is evident the Classifier is not foolproof. Furthermore, OpenAI mentions that it’s unreliable on short texts below 1,000 characters.
“Our classifier has a number of important limitations. It should not be used as a primary decision-making tool, but instead as a complement to other methods of determining the source of a piece of text.”OpenAI
The tool has some other limitations, including support for only English inputs, difficulty in predictable texts, and being poorly calibrated outside its training data. For all those reasons and more, this new AI Classifier should not be used as a primary decision making-tool just yet. However, as time passes, we can expect improvement as more data is fed to it.
How Does OpenAI Classifier Fare in Our Testing
OpenAI’s AI Text Classifier (website link) is an online tool that’s free to use. So, we decided to take it for a short spin, and it performs as well as you would expect at this stage. I tested out varying articles of moderate length from our website. These included news pieces and feature articles. After pasting the text and running the tool, we got the response “The Classifier considers the text to be very unlikely AI-generated.”
To test how well it works, I had ChatGPT write up a short story and used the Classifier to see if it could identify the story was written by its own AI sibling. Thankfully, it rendered the response that the text was “likely AI-generated”. I tried the same with some other texts like The Merchant of Venice by Shakespeare and the AI detected unlikely AI involvement.
As mentioned above, the OpenAI Classifier should get better as its dataset increases. However, even until that happens, you are free to use the tool and have fun with it. So what are your thoughts on AI text detectors? Do you feel they’re a necessity in 2023? Drop your two cents in the comments below.
