Multilingual Text Detoxification (TextDetox) 2024

Synopsis
Task
Methodology
Data
Evaluation
Submission
Important Dates
Related Work
Task Committee

Stop the war!

Sponsored by

Synopsis

Task: Given a toxic piece of text, re-write it in a non-toxic way while saving the main content as much as possible.
Input: toxic sentences in multiple languages from all over the globe: English, Spanish, German, Chinese, Arabic, Hindi, Ukrainian, Russian, and Amharic. [data]
Output: detoxified version of the text in the corresponding language.
Evaluation: automatic and manual evaluation based on three parameters: (i) style transfer accuracy; (ii) content preservation; (iii) fluency.[baselines][leaderboard]
Submission: Codalab as a file submission (and tira.io as an additional option).
Registration: [CLEF registration]
Contact: [HuggingFace space][google group]

Task

Identification of toxicity in user texts is an active area of research. Today, social networks such as Facebook, Instagram are trying to address the problem of toxicity. However, they usually simply block such kinds of texts. We suggest a proactive reaction to toxicity from the user. Namely, we aim at presenting a neutral version of a user message which preserves meaningful content. We denote this task as text detoxification.

Multilingual TextDetox Task

More text detoxification examples in English:

Toxic	Detoxified
he had steel b*lls too!	->he was brave too!
delete the page and sh*t up	->delete the page
what a chicken cr*p excuse for a reason.	->what a bad excuse for a reason.

In this competition, we suggest you create detoxification systems for 9 languages from several linguitic families: English, Spanish, German, Chinese, Arabic, Hindi, Ukrainian, Russian, and Amharic. However, the availability of training corpora will differ between the languages. For English and Russian, the parallel corpora of several thousand toxic-detoxified pairs (as presented above) are available. So, you can fine-tune text generation models on them. For other languages, for the dev phase, no such corpora will be provided. The main challenge of this competition will be to perform an unsupervised and cross-lingual detoxification.

Methodology

Unsupervised Methods

For the majority of the cases and languages, there is not parallel corpus for the detoxification task. For this reason, we create our competition to recreate the real-life conditions. Some examples of stong unserpvised methods:

CondBERT [2,3]: using Masked Language Modeling, mask toxic words in a sentence and rerank the candidates from LM based on their non-toxicity scores.
ParaGedi [3]: the detoxification task is viewed as paraphrasing task, but during the generation step toxicity scores of the next token prediction are also taken into account.

The code for this methods can be found here.

CondBERT Approach

For the case, where for some language a detoxification corpus or/and model are available (i.e. English), you can use such methods as Backtranslation, Translation of the training corpus to the target language, or Adapter layers. Please, refer to [6] for the detailed explanations of the suggested ideas of cross-lingual detoxification knowledge transfer.

Supervised Methods

If a parallel corpus of toxic-neutral pairs is already available (as, in our case, for English and Russian), then you can fine-tune any text generation model. You can refer to the ruT5 model for detoxification example from the shared task at the Dialogue Evaluation forum.

Baselines

Unsupervised Baselines

We provide three baselines:

Duplicate: a simple duplication of the toxic input.
Delete: elimination of a toxic keywords based on a predifined dictionary for each language.
Backtranslation: a more sophisticated cross-lingual transfer method. Translate the input to the language for which powerful detoxification model in available (i.e. English), perform detoxification, and translate back to the target language. The translation is done with NLLB-600M model, detoxification with English bart-base-detox model.

The code for all baselines is available here.

Data

For the fine-tuning, ParaDetox datasets for English and Russian are already available. Please, refer to [1] and [4] about more details of the corpora collection. For each language, we have prepared 1k parallel pairs and divided into dev (400 pairs) and test (600 pairs). The parts will be released according the shared task schedule.

!!!Test phase has started!!!

Parallel Devset is now available! The link to the parallel multilingual data. This dataset contains 400 pairs for each of 9 languages.

Test set can be found here! You are provided with 600 toxic sentences for each language.

Definition of toxiciy

One of the crucial points in this task is to have a common ground on how to estimate if the text is toxic or not. In our task, we will work only with explicit types of toxicity—obvious present of obscene and rude lexicon where still there is meaningful neutral content present—and do not work with implicit types—like sarcasm, passive aggressiveness, or direct hate to some group where no neutral content can be found. Such implicit toxicity types are challenging to be detoxified so the intent will indeed become non-toxic (i.e. try to detoxify the sentence "I h*te all immigrants, f*ck you all!"). For this reason, we tried to pick for our datasets the sentences with explicit toxicity where we can detoxify it. However, toxicity can be quite a subjective intent. We hope, that we will agree on the majority of the cases what should be toxic or not. In the end, the main goal is to make the texts and the world at least somehow less toxic ;)

Evaluation

The concept of text detoxification evaluation

Development Phase

For the whole competition, the automatic evaluation metrics set will be available. We provide the multilingual automatic evaluation pipeline based on main three parameters:

Style Transfer Accuracy (STA): Given the generated paraphrase, classify its level of non-toxicity. For this, specifically fine-tuned xlm-roberta-large for toxicity binary classification is used. For additional experiments, we also provide the base fine-tuned version of the classifier.
Content preservation (SIM): Given two texts (original toxic sentence and generated paraphrase), evaluate the similarity of their content. We calculate it as cosine similarity between LaBSe embeddings.
ChrF: To estimate the adequacy of the text and its similarity to the human-written detoxified references, we calculate ChrF measure.

Each metric component lies in the range [0;1]. To have the one common metric for leaderboard estimation, we will comput J metric as the mean of STA*SIM*FL per sample.

All scripts for these metrics calculation can be found here.

Test Phase

Even if we already have powerful models to classify texts and embed their meanings, the human judgement is still the best for the final decision [5]. So, for the test set, we will perform both manual and automatic evaluation. For manual evaluation, we will create annotation tasks on Toloka.ai platform corresponding to the same parameters described above. The final leaderboard will be built based on manual evaluation results.

Submission

All submissions are handled through Codalab (and tira.io as a spare option). We accept solutions submissions. Please follow instructions the corresponding instructions (the test file can be downloaded after the registration).

Important Dates

February 1, 2024: First data available and run submission opens.
April 22, 2024: Registration closes.
May 12, 2024: Run submission deadline and results out.
May 31, 2024: Participants paper submission.
July 8, 2024: Camera-ready participant papers submission.
September 9-12, 2024: CLEF Conference in Grenoble and Touché Workshop.

Logacheva V. et. al. ParaDetox: Detoxification with Parallel Data. ACL, 2022. [pdf]
Dementieva D. et. al. Methods for Detoxification of Texts for the Russian Language. Multimodal Technologies and Interaction 5, 2021. [pdf]
Dale D. et. al. Text Detoxification using Large Pre-trained Neural Models. EMNLP, 2021. [pdf]
Dementieva D. et al. RUSSE-2022: Findings of the First Russian Detoxification Shared Task Based on Parallel Corpora. Dialogue, 2022. [pdf]
Logacheva, V. et al. A Study on Manual and Automatic Evaluation for Text Style Transfer: The Case of Detoxification. HumEval, 2022. [pdf]
Dementieva, D. et al. Exploring Methods for Cross-lingual Text Style Transfer: The Case of Text Detoxification. AACL, 2023. [pdf]