Multilingual Text Detoxification 2024

Synopsis

  • Task: Given a toxic piece of text, re-write it in a non-toxic way while saving the main content as much as possible.
  • Input: toxic sentences in several languages: English, Russian, Ukrainian, German, Chinese, Amharic, and Arabic.
  • Output: non-toxic version in the corresponding language.
  • Evaluation: automatic and manual evaluation based on three parameters: (i) style transfer accuracy; (ii) content preservation; (iii) fluency.
  • Submission: tira.io as a software submission.

Task

Identification of toxicity in user texts is an active area of research. Today, social networks such as Facebook, Instagram are trying to address the problem of toxicity. However, they usually simply block such kinds of texts. We suggest a proactive reaction to toxicity from the user. Namely, we aim at presenting a neutral version of a user message which preserves meaningful content. We denote this task as *detoxification*.

Detoxification can be solved with Text Style Transfer (TST) methods. Before for detoxification, only unsupervised methods were used. Meaning, there was no parallel corpora, only non-parallel datasets with toxic and non-toxic labels. Recenly, the first parallel detoxification corporus (with pairs toxic-neutral text) was introduced for English (Paradetox) and then for Russian (RUSSE-2022). However, the tranfer of detoxification knowledge to a new language is still challenging.

In this competition, we suggest you to create detoxification systems for new languages—Ukrainian, German, Chinese, Amharic, and Arabic—for those there is no parallel detoxification corpus. Meaning, you need to solve the task of cross-lingual text detoxification transfer.

Data

ParaDetox datasets for English and Russian are already available. For each new language, we will provide 1k parallel pairs for development fase and 100 toxic sentences for the test set.

Definition of toxiciy. One of the crucial points in this task is to have a common ground on how to estimate if the text is toxic or not.

Evaluation

For the development set, the automatic evaluation metrics set will be available. For the test set, both automatic and manual evaluations will be hold. The final leaderboard will be built based on manual evaluation results.

Both automatic and manual evaluation will be based on the main three parameters:

  • Style Transfer Accuracy (STA): Given the generated paraphrase, the annotators should classify it into one of the classes – toxic or neutral.
  • Content preservation (SIM): Given two texts (original toxic sentence and generated paraphrase) the annotators should evaluate the similarity of their content and mark them as similar or dissimilar.
  • Fluency task (FL) The output should be no less fluent then the input sentence.

To have the one common metric for leaderboard estimation, we will comput J metric as the mean of STA*SIM*FL per sample.

For automatic evaluation, the setup for each language will look like follow: (i) STA: binary classifier; (ii) SIM: cosine similarity of LaBSE embeddings; (iii) FL: binary classifier or perplexity. All scripts for these metrics calculation will be provided.

For manual evaluation, we will create annotation tasks on Toloka.ai platform. Then, the final leaderboard will be published on this page.

Submission

All submissions are handled through tira.io. We encourage all participants to use software submissions (via docker) for the sake of reproducibility. In a software submission you upload a docker image, which is then built and executed on the test data on tira's servers. Please follow tira's extensive documentation (and forums) for instructions.

  • Logacheva V. et. al. ParaDetox: Detoxification with Parallel Data. ACL, 2022. [pdf]
  • Dementieva D. et. al. Methods for Detoxification of Texts for the Russian Language. Multimodal Technologies and Interaction 5, 2021. [pdf]
  • Dale D. et. al. Text Detoxification using Large Pre-trained Neural Models. EMNLP, 2021. [pdf]
  • Dementieva D. et al. RUSSE-2022: Findings of the First Russian Detoxification Shared Task Based on Parallel Corpora. Dialogue, 2022. [pdf]

Task Committee