Voight-Kampff Generative AI Detection 2025

Synopsis

  • Subtask 1: Given a (potentially obfuscated) text, decide whether it was written by a human or an AI.
  • Subtask 2: Given a document collaboratively authored by human and AI, classify the extent to which the model assisted.
  • Important dates: May 23, 2025 (software submission), May 30, 2025 (participant notebook submission)
  • Data: Human and machine texts [download task 1] [download task 2]
  • Evaluation Measures: F1, C@1, FPR, FNR
  • Baselines: SVM, Binoculars, RoBERTa [code task 1] [code task 2]

Task Overview

The Generative AI Authorship Verification Task @ PAN is split into two subtasks [subtask 1, subtask 2]. Participants can submit their systems to either of them or both. Task 1 focuses on the robustness and sensitivity of detection systems, Task 2 focuses on the degree to which a mixed-authorship text is human- or machine-authored. The two tasks have individual datasets.

Subtask 1: Voight-Kampff AI Detection Sensitivity

Subtask 1 is a binary AI detection task in that participants are given a text and have to decide whether it was machine-authored (class 1) or human-authored (class 0). However, we introduced a twist: The LLMs were instructed to change their style and mimic a specific human author. Furthermore, the test set will contain several surprises such as new models or unknown obfuscations to test the robustness of the classifiers (however, texts will be from the same domain).

As in the previous year, the Voight-Kampff AI detection Task @ PAN is organized in collaboration with the Voight-Kampff Task @ ELOQUENT Lab Lab in a builder-breaker style. PAN participants will build systems to tell human and machine apart, while ELOQUENT participants will investigate novel text generation and obfuscation methods for avoiding detection.

Data

The dataset is available via Zenodo. Please register first at Tira and then request access on Zenodo using the same email address. The dataset contains copyrighted material and may be used only for research purposes. No redistribution allowed.

The training and validation dataset is provided as a set of newline-delimited JSON files. Each file contains a list of texts, written either by a human or a machine. The file format is as follows:

{"id": "a6c8018e-d22c-4d6e-b5e3-0c0a65682a6a", "text": "...", "model": "human", "label": 0, "genre": "essays"}
{"id": "f1a26761-ca2a-43e9-890d-80dcb3058364", "text": "...", "model": "gpt-4o", "label": 1, "genre": "essays"}
...
A "label" of 0 means human-written, 1 is ai-written. "genre" is for informational purposes only and can be either "essays", "news", or "fiction". Texts with "genre": "news" are sampled from last year's dataset (but with a few additions, such as GPT-4o). So if you want to reuse last year's dataset, be aware that some texts will be duplicates! The test dataset will have the same format, but with only the "id" and "text" columns.

Submission

Participants will submit their systems as Docker images through the Tira platform. It is not expected that submitted systems are actually trained on Tira, but they must be standalone and runnable on the platform without requiring contact to the outside world (evaluation runs will be sandboxed).

The submitted software must be executable inside the container via a command line call. The script must take two arguments: an input file (an absolute path to the input JSONL file) and an output directory (an absolute path to where the results will be written):

Within Tira, the input file will be called dataset.jsonl, so with the pre-defined Tira placeholders, your software should be invoked like this:

$ mySoftware $inputDataset/dataset.jsonl $outputDir

Within $outputDir, a single (!) file with the file extension *.jsonl must be created with the following format:

{"id": "bea8cccd-0c99-4977-9c1b-8423a9e1ed96", "label": 1.0}
{"id": "a963d7a0-d7e9-47c0-be84-a40ccc2005c7", "label": 0.2315}
...

For each test case in the input file, an output line must be written with the ID of the input text pair and a confidence score between 0.0 and 1.0. A score < 0.5 means that the text is believed to be human-authored. A score > 0.5 means that it is likely machine-written. A score of exactly 0.5 means the case is undecidable. Participants are encouraged to answer with 0.5 rather than making a wrong prediction. You can also give binary score (0 and 1) if your detector does not output class probabilities.

All test cases must be processed in isolation without information leakage between them! Even though systems may be given an input file with multiple JSON lines at once for reasons of efficiency, these inputs must be processed and answered just the same as if only a single line were given. Answers for any one test case must not depend on other cases in the input dataset!

Evaluation

tba

Baselines

tba

Subtask 2: Human-AI Collaborative Text Classification

In subtask 2, we focus on Human-AI Collaborative Text Classification, where the goal is to categorize documents that have been co-authored by humans and LLMs. Specifically, we aim to classify texts into six distinct categories based on the nature of human and machine contributions:

  1. Fully human-written: The document is entirely authored by a human without any AI assistance.
  2. Human-initiated, then machine-continued: A human starts writing, and an AI model completes the text.
  3. Human-written, then machine-polished: The text is initially written by a human but later refined or edited by an AI model.
  4. Machine-written, then machine-humanized (obfuscated): An AI generates the text, which is later modified to obscure its machine origin.
  5. Machine-written, then human-edited: The content is generated by an AI but subsequently edited or refined by a human.
  6. Deeply-mixed text: The document contains interwoven sections written by both humans and AI, without a clear separation.

Accurately distinguishing between these categories will enhance our understanding of human-AI collaboration and help mitigate the risks associated with synthetic text.

Data

The dataset for Task 2 can be downloaded from Zenodo. More information and baselines can be found in our GitHub repository.
  • Multi-domain documents (academic, journalism, social media)
  • Human-written and machine-generated samples (GPT-4, Claude, PaLM)
  • Collaborative texts with annotation layers for human/machine contributions
  • Multiple languages supported (English, Spanish, German)

Dataset label distribution:

Label Category Train Dev
Machine-written, then machine-humanized 91,232 10,137
Human-written, then machine-polished 95,398 12,289
Fully human-written 75,270 12,330
Human-initiated, then machine-continued 10,740 37,170
Deeply-mixed text (human + machine parts) 14,910 225
Machine-written, then human-edited 1,368 510
Total 288,918 72,661

Submission

tba

Evaluation

tba

Baselines

  • Zero-shot detectors: DetectGPT, Binoculars
  • Fine-tuned LLMs: RoBERTa-base, DeBERTa-v3
  • Ensemble methods with stylometric features

Code for the task baseline can be found in our GitHub repository.

Task Committee