Decoding the AI's Gaze: Unraveling ChatGPT's Evaluation of Poetic Creativity

As artificial intelligence (AI) technology advances, it becomes increasingly challenging to distinguish between human-written and AI-generated poetry. In this explorative study, we employ a novel approach to the Turing Test, traditionally used to evaluate a machine's ability to exhibit human-like intelligence. We use the Reverse Turing Test to classify poems as either human-written or AI-generated, with the AI (ChatGPT, GPT-4) serving as the evaluator. The AI analyzed 18 poems, half of which were in their original form (authored by classic poets), and half were with AI-generated continuations created using a pretrained GPT-3 model. However, despite ChatGPT's extensive training on a vast textual database, its classification performance on human and AI-generated poems did not surpass random guessing. This raises questions about the AI's ability to accurately distinguish between the two, particularly with original texts from renowned classic authors. The qualitative analysis reveals significant disparities in the evaluation of human-classified versus AI-classified poems, demonstrating a bias where the same markers, such as consistency, coherence, and fluency, are viewed positively in poems classified as human-written but negatively in poems classified as AI-generated. The results indicate a consistent pattern in the assessment criteria for both human- and AI-classified poems, often neglecting their actual textual characteristics. This study not only highlights the current challenges in differentiating between human and AI generated poetry but also provides insights into the cues and heuristics AI uses for such classifications.