Deconstructing Digital Discourse: A Deep Dive into Distinguishing LLM-Powered Chatbots from Human Language - Rao, Jiarui; Zhang, Qian (Journal of Theory and Practice in Education and Innovation, 2025)
In recent years, chatbots powered by Large Language Models (LLMs) have garnered significant attention in the field of artificial intelligence. These models are sophisticated natural language processing systems trained using advanced deep learning techniques. The development process involves several crucial steps. Initially, the dataset is visualized and analyzed to understand its characteristics. This is followed by text preprocessing to clean and prepare the data for training. Subsequently, the language for the chatbot is generated and further processed using Deberta v3. Finally, a machine learning classifier is employed to distinguish between text generated by the chatbot and natural human language. The evaluation results indicate that the model achieves an accuracy of 85%, a precision of 60%, a recall of 62%, and an F1 score of 0.61. The high accuracy demonstrates the model's capability to differentiate between chatbot-generated text and natural language. However, the precision and recall values, both close to 60%, still suggest a significant degree of ambiguity. This makes it relatively easy for chatbot-generated text to be confused with natural human language.