Text Watermarking 2026

Synopsis

• Task: Insert a watermark into a given text. Then, after we have attacked the text, detect the inserted watermark.
• Registration: [CLEF labs] [Tira]
• Important dates:
  • May 07, 2026 May 21, 2026: software submission [baseline  – together with step-by-step submission instructions ]
  • May 28, 2026: participant notebook submission [template] [submission  – select "Stylometry and Digital Text Forensics (PAN)" ]
• Data: Political speeches [download]
• Evaluation Measures: Balanced Accuracy, BLEU, BERTScore
• Baselines: [code].

Task Overview

In the Text Watermarking task, participants are given a text and must insert a watermark into it. After submitting the watermark system, including a watermark detection algorithm, through TIRA, the watermark system is run on the test dataset. The watermarked texts are then subjected to various attacks. The objective is to detect the watermark after the text has been attacked, thereby demonstrating its robustness against an attacker.

The task is structured as follows:

1. Develop a text watermarking system, that inserts a watermark into a text. You can use the provided training dataset for this purpose.
2. Submit your watermarking system, including the watermark detection, through the Tira platform.
3. We will run your submitted watermarking system on our test dataset.
4. We will then carry out attacks of varying severity on the watermarked texts from the test dataset.
5. We will run your watermark detection system on the attacked texts to evaluate its performance in detecting watermarks.

The watermarked text must remain semantically close to the original. Watermarked texts that deviate significantly in meaning from the original will be penalized during evaluation.

Data

The dataset is available via Zenodo.

The training dataset is provided as a set of newline-delimited JSON files. The file format is as follows:

{"id": "9a28d103-7b2e-43da-b511-2efb5f91975f", "text": "..."}
{"id": "4fcae8a8-f5e6-4ccd-9dae-10b609a47cfc", "text": "..."}
...

The test dataset will have the exact same format.

Submission

Participants will submit their systems as Docker images through the Tira platform. Please have a look at the baseline for step-by-step submission instructions. It is not expected that submitted systems are actually trained on Tira, but they must be standalone and runnable on the platform without requiring contact to the outside world (evaluation runs will be sandboxed).

Important: As contact to the outside world is not possible in the TIRA environment, submissions cannot use online services such as ChatGPT or other online LLMs.

The submitted software must be executable inside the container via a command line call. The script must take two arguments: an input file (an absolute path to the input JSONL file) and an output directory (an absolute path to where the results will be written):

Within Tira, the input file will be called dataset.jsonl, so with the pre-defined Tira placeholders, your software should be invoked like this:

$ mySoftware $inputDataset/dataset.jsonl $outputDir

Within $outputDir, a single (!) file with the file extension *.jsonl must be created with the following format:

{"id": "bea8cccd-0c99-4977-9c1b-8423a9e1ed96", "label": 1.0}
{"id": "a963d7a0-d7e9-47c0-be84-a40ccc2005c7", "label": 0.0}
...

Evaluation

Systems will be evaluated with the following metrics:

• Overall score, here called Text Watermarking Fidelity (TWF). TWF = max(BLEU, BERTScore) · Balanced Accuracy
• Balanced Accuracy: The average of true positive rate and true negative rate.
• BLEU: Syntactic similarity of the watermarked text to the original text. Here, it is computed at sentence level and averaged over the full text.
• BERTScore: Semantic similarity between the watermarked text and the original text. Here, it is computed at sentence level and averaged over the full text.
• In addition, the confusion matrix is reported for calculating true/false positive/negative rates.

The evaluator for the task will output the above measures as JSON like so:

{
    "twf": 0.878,
    "balanced_accuracy": 0.974,
    "bleu": 0.901,
    "bert-score": 0.85,
    "confusion": [
        [
            1211,
            66
        ],
        [
            27,
            2285
        ]
    ]
}

Baselines

The baseline is published on GitHub.

More information on how to install and run the baseline can be found in the README on GitHub.

Leaderboard

#	Team	Software	TWF	BA	BLEU	BERTScore
1	yeyang	mild-longitude	0.818	0.824	0.948	0.992
2	aimoment	matching-setter	0.775	0.785	0.904	0.988
3	team	flavorful-classifier	0.762	0.773	0.857	0.987
4	JRC-GENESIS	gravitational-albariño	0.761	0.785	0.642	0.969
5	aimoment	interior-home	0.734	0.753	0.789	0.975
6	alex-csed-3n	binding-yeoman	0.727	0.819	0.071	0.887
7	JRC-GENESIS	direct-bot	0.686	0.720	0.572	0.952
	KGW Baseline		0.633	0.720	0.014	0.880