Generative AI Detection 2025

Synopsis

• Task: tba.
• Submission: Deployment on TIRA [submit]
• Input: tba. [download]
• Evaluation Measures: tba. [code]
• Baselines: tba. [code]
• Evaluation: tba. [code]

Task

Develop models to detect and classify human-AI collaborative texts:

Subtask 1: AI Detection Sensitivity

• Detect AI-generated text with/without obfuscation
• Binary classification: Human vs Machine

Subtask 2: Human-AI Collaborative Text Classification

Given a document collaboratively authored by humans and models, our goal is to classify it into one of the following six categories:

• (1) Fully human-written
• (2) Human-initiated, then machine-continued
• (3) Human-written, then machine-polished
• (4) Machine-written, then machine-humanized (obfuscated)
• (5) Machine-written, then human-edited
• (6) Deeply-mixed text: where some parts are written by a human and some are generated by a machine.

Data

• Multi-domain documents (academic, journalism, social media)
• Human-written and machine-generated samples (GPT-4, Claude, PaLM)
• Collaborative texts with annotation layers for human/machine contributions
• Multiple languages supported (English, Spanish, German)

Evaluation

• Macro-F1
• Robustness testing against novel obfuscation techniques

Submission

tba

Baselines

• Zero-shot detectors: DetectGPT, Binoculars
• Fine-tuned LLMs: RoBERTa-base, DeBERTa-v3
• Ensemble methods with stylometric features

Results

tba

Related Work

• Bevendorff, Janek, Xavier Bonet Casals, Berta Chulvi, Daryna Dementieva, Ashaf Elnagar, Dayne Freitag, Maik Fröbe, et al. 2024. Overview of PAN 2024: Multi-Author Writing Style Analysis, Multilingual Text Detoxification, Oppositional Thinking Analysis, and Generative AI Authorship Verification: Extended Abstract. In Lecture Notes in Computer Science, 3-10. Lecture Notes in Computer Science. Cham: Springer Nature Switzerland.
• Uchendu, Adaku, Thai Le, Kai Shu, and Dongwon Lee. 2020. Authorship Attribution for Neural Text Generation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), 8384-95. Online: Association for Computational Linguistics.
• Jakesch, Maurice, Jeffrey T. Hancock, and Mor Naaman. 2023. Human Heuristics for AI-Generated Language Are Flawed. Proceedings of the National Academy of Sciences of the United States of America 120 (11): e2208839120.
• Hans, Abhimanyu, Avi Schwarzschild, Valeriia Cherepanova, Hamid Kazemi, Aniruddha Saha, Micah Goldblum, Jonas Geiping, and Tom Goldstein. 2024. Spotting LLMs with Binoculars: Zero-Shot Detection of Machine-Generated Text. arXiv [Cs.CL].
• Su, Jinyan, Terry Yue Zhuo, Di Wang, and Preslav Nakov. 2023. DetectLLM: Leveraging Log Rank Information for Zero-Shot Detection of Machine-Generated Text. arXiv [Cs.CL].
• Mitchell, Eric, Yoonho Lee, Alexander Khazatsky, Christopher D. Manning, and Chelsea Finn. 2023. DetectGPT: Zero-Shot Machine-Generated Text Detection Using Probability Curvature. arXiv [Cs.CL].
• Bao, Guangsheng, Yanbin Zhao, Zhiyang Teng, Linyi Yang, and Yue Zhang. 2023. Fast-DetectGPT: Efficient Zero-Shot Detection of Machine-Generated Text via Conditional Probability Curvature. arXiv [Cs.CL].
• Koppel, Moshe, and Jonathan Schler. 2004. Authorship Verification as a One-Class Classification Problem. In Proceedings, Twenty-First International Conference on Machine Learning, ICML 2004, 489-95.
• Bevendorff, Janek, Benno Stein, Matthias Hagen, and Martin Potthast. 2019. Generalizing Unmasking for Short Texts. In Proceedings of the 2019 Conference of the North, 654-59. Stroudsburg, PA, USA: Association for Computational Linguistics.
• Sculley, D., and C. E. Brodley. 2006. Compression and Machine Learning: A New Perspective on Feature Space Vectors. In Data Compression Conference (DCC'06), 332-41. IEEE.
• Halvani, Oren, Christian Winter, and Lukas Graner. 2017. On the Usefulness of Compression Models for Authorship Verification. In ACM International Conference Proceeding Series. Vol. Part F1305. Association for Computing Machinery. https://doi.org/10.1145/3098954.3104050.
• Uchendu, Adaku, Zeyu Ma, Thai Le, Rui Zhang, and Dongwon Lee. 2021. TURINGBENCH: A Benchmark Environment for Turing Test in the Age of Neural Text Generation. In Findings of the Association for Computational Linguistics: EMNLP 2021, 2001-16. Stroudsburg, PA, USA: Association for Computational Linguistics.
• Schuster, Tal, Roei Schuster, Darsh J. Shah, and Regina Barzilay. 2020. The Limitations of Stylometry for Detecting Machine-Generated Fake News. Computational Linguistics 46 (2): 499-510.
• Sadasivan, Vinu Sankar, Aounon Kumar, Sriram Balasubramanian, Wenxiao Wang, and Soheil Feizi. 2023. Can AI-Generated Text Be Reliably Detected? arXiv [Cs.CL].
• Ippolito, Daphne, Daniel Duckworth, Chris Callison-Burch, and Douglas Eck. 2020. Automatic Detection of Generated Text Is Easiest When Humans Are Fooled. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 1808-22. Stroudsburg, PA, USA: Association for Computational Linguistics.

Task Committee