Oppositional thinking analysis: Conspiracy theories vs critical thinking narratives

Synopsis

Conspiracy theories are complex narratives that attempt to explain the ultimate causes of significant events as cover plots orchestrated by secret, powerful, and malicious groups [1]. A challenging aspect of identifying conspiracy with NLP models [2] stems from the difficulty of distinguishing critical thinking from conspiratorial thinking in automatic content moderation. This distinction is vital because labeling a message as conspiratorial when it is only oppositional could drive those who were simply asking questions into the arms of the conspiracy communities.

At PAN 2024 we aim at analyzing texts that reflect oppositional thinking and contain either conspiracy or critical narratives. The task will address two new challenges for the NLP research community: (1) to distinguish the conspiracy narrative from other oppositional narratives that do not express a conspiracy mentality (i.e., critical thinking); and (2) to identify in online messages the key elements of a narrative that fuels the intergroup conflict in oppositional thinking. To this end we provide two text corpora, one English and one Spanish, and we propose two sub-tasks:

1. Distinguishing between critical and conspiracy texts (subtask 1):

A binary classification task differentiating between (1) critical messages that question major decisions in the public health domain, but do not promote a conspiracist mentality; and (2) messages that view the pandemic or public health decisions as a result of a malevolent conspiracy by secret, influential groups.

• Input: Set of texts, each associated with one of the two categories: CONSPIRACY, CRITICAL
• Official evaluation metric: MCC [3]
• Baselines: BERT classifier [4]

2. Detecting elements of the oppositional narratives (subtask 2):

A token-level classification task aimed at recognizing text spans corresponding to the key elements of oppositional narratives. Since conspiracy narratives are a special kind of causal explanation, we developed a span-level annotation scheme that identifies the goals, effects, agents, and the groups-in conflict in these narratives.

• Input: Set of texts, each text accompanied with a (possibly empty) list of span annotations. Each annotation corresponds to a narrative element, and is described by its borders (start and end characters), and its category. There are six distinct span categories: AGENT, FACILITATOR, VICTIM, CAMPAIGNER, OBJECTIVE, NEGATIVE_EFFECT
• Official evaluation metric: macro-averaged span-F1 [5]
• Baseline: BERT-based multi-task token classifier (separate classification heads, common transformer backbone) [6]

GitHub repository with utilities, baselines, and additional instructions and guidelines can be found here

Task

Conspiracy Theories are complex narratives that attempt to explain the ultimate causes of significant events as cover plots orchestrated by secret, powerful, and malicious groups [1]. Automatic detection of CTs in text has recently gained popularity [2, 7, 8, 9]. The problem is commonly framed as binary classification, with fine-grained approaches corresponding to multi-label or multi-class classification. Two recent MediaEval challenges [8, 9] of coarse- and fine-grained classification of conspiratorial text [8, 9] led to a number of approaches demonstrating that the state-of-art architecture is a multi-task classifier [10, 11, 12] based on domain-specific CT-BERT model [12]. An LLM-based approach was also attempted [13].

However, existing approaches do not distinguish between critical and conspiratorial thinking. This distinction is important because labeling a text as conspiratorial when it is, in fact, oppositional to mainstream views, could potentially lead those who were simply asking questions closer to conspiracy communities. As several authors from the field of social sciences suggest, a fully-fledged conspiratorial worldview is the final step in a progressive "spiritual journey" that sets out by questioning social and political orthodoxies [1, 14, 15]. Additionally, recent research [16] has shown that the level of interaction with conspiracist users is the most important feature for predicting whether or not users join conspiracy communities. These insights have an important implication for automatic content moderation: if models do not differentiate between critical and conspiratorial thinking, there is a high risk of pushing people toward conspiracy communities.

Another important gap in the computational analysis of conspiratorial texts fails to address is the role that intergroup conflict (IGC) [17] plays in these narratives. Intergroup conflict is a way of framing events by emphasizing the hostility between groups, typically by using "us versus them" narrative, and by fueling the perceived injustice and threat to the group. The increasing potentially violent involvement of conspiracist communities in political processes suggests that one of the purposes of CTs is to enforce IGC and coordinate action [18]. Therefore, tools that enable an IGC-based analysis of conspiratorial texts could offer valuable insights for content moderation.

Motivated by the described issues, we propose a novel annotation scheme that distinguishes between conspiracy and critical texts, and defines important categories of oppositional narrative. In addition to the standard elements of conspiracy narratives such as agents (conspirators) and victims, the proposed scheme identifies the following categories: “facilitators” (collaborators of the agents, such as the media) and “campaigners” (those that unmask the conspiracy agenda). These types of actors are “key players” in IGC: the facilitators are tangible targets with whom real conflict is possible (in contrast to abstract agents such as secret groups), and the campaigners are those that show their opposition to the facilitators and try to persuade the victims to join their cause.

We focus on oppositional texts from the Telegram platform related to the COVID-19 pandemic, and construct English and Spanish corpora annotated with the described labeling schemes. This enables the NLP community to tackle two new tasks related to the two previously described phenomena: the binary classification task of distinguishing between conspiratorial and critical texts, and the task of detecting the elements of the oppositional narrative.

Award

We are pleased to announce that the best-performing team at the 11th International Competition on Author Profiling will be awarded 300,- Euro sponsored by Symanto.

This year, the winner of the task is:

• Angelo Maximilian Tulbure and María Coll Ardanuy (tulbure team) from Politecnico di Milano and Universitat Politècnica de València

Data [download]

The participants will work with a JSON file that contains all the texts in the train dataset, and their annotations. Each text corresponds to a dictionary that contains the ID, tokenized text, the binary category, and the span annotations. Span annotations are a list of dictionaries, each corresponding to an annotated span and containing the span’s category and text, designated by the start and end characters. This is an example of a JSON dictionary corresponding to a single, fully annotated, text:

                 {
                	"id": "91221",
                	"text": "\" Scientism backed ... not \" \" science - backed \" \" . There is nothing scientific about the Covid or childhood vaccine quackery that is slowly but surely killing , maiming and neurologically injuring the next generation — just as the Clintons and other Moloch worshippers want . \" ",
                	"category": "CONSPIRACY",
                	"annotations": [
                  	{
                    	"span_text": "that is slowly but surely killing , maiming and neurologically injuring the next generation",
                    	"category": "NEGATIVE_EFFECT",
                    	"start_char": 128,
                    	"end_char": 219,
                  	},
                  	{
                    	"span_text": "the next generation",
                    	"category": "VICTIM",
                    	"start_char": 200,
                    	"end_char": 219,
                  	},
                  	{
                    	"span_text": "the Clintons",
                    	"category": "AGENT",
                    	"start_char": 230,
                    	"end_char": 242,
                  	},
                  	{
                    	"span_text": "other Moloch worshippers",
                    	"category": "AGENT",
                    	"start_char": 247,
                    	"end_char": 271,
                  	}
                	]
                  }

            

At test time, the participants will receive text data in the above format. For each text, only the “id” and the “text” fields will be provided. The required output is a JSON file in the same format, with each text designated with an ID, and annotated with the text category, and the annotations. More details about the data, and the data utilities can be found in the task’s github repository.

Submission

The test set has been released [download].

The submission must be a ZIP file with a folder named as your team. The ZIP file must contain a JSON file with each of the subtasks and languages you want to participate in. The name of each JSON file must follow the structure below:

[team]_[task]_[lang].json

For instance, for team "rangel", you will send a file whose name will be rangel.zip with the following structure and files:

            rangel/
            |----rangel_task1_en.json
            |----rangel_task1_es.json
            |----rangel_task2_en.json
            |----rangel_task2_es.json
            

The content of the JSON file depends on the subtask:

Subtask 1:

                [
                  {
                    "id": "6742",
                    "category": "CRITICAL"
                  },
                  {
                    "id": "10596",
                    "category": "CONSPIRACY"
                  },
                ....
                ]
            

Subtask 2:

                [
                  {
                    "id": "6742",
                    "annotations": [
                      {
                        "start_char": 255,
                        "end_char": 360,
                        "category": "VICTIM"
                      },
                      {
                        "start_char": 89,
                        "end_char": 111,
                        "category": "VICTIM"
                      },
                      {
                        "start_char": 204,
                        "end_char": 271,
                        "category": "OBJECTIVE"
                      },
                      {
                        "start_char": 194,
                        "end_char": 275,
                        "category": "FACILITATOR"
                      }
                    ]
                  },
                ...
                ]
            

The ZIP file has to be sent to: kico (dot) rangel (at) gmail (dot) com

Please, use the following as a template for the title of the email where you substitute the placeholder for your team name and run number:

[PAN'24 - Oppositional Thinking Analysis] Evaluation of [team]'s run[1|2]

We will send you the obtained results as soon as we evaluate them, as well as we will contact you in case of technical errors.

IMPORTANT: Remember, you can submit up to two runs and we will keep the one with the highest score unless you tell us the contrary. In case of doubt, we will ask you to let us know which one you want to use for the final ranking.

Evaluation

The official evaluation metric for subtask 1 (critical vs. conspiracy classification) is MCC [3], while the official metric for subtask 2 (span-level detection of narrative elements) is macro-averaged span-F1 [5]. For subtask 1 we will also provide binary F1 scores for each of the classes, and for subtask 2 we will provide per-category span-F1 scores. For each of the task languages, English and Spanish, a separate ranking list will be maintained for each task.

We provide two hard baselines for these tasks. For subtask 1, the baseline is a standard BERT [4] classifier. For subtask 2, the baseline is a BERT-based multi-task token classifier (separate classification heads, common transformer backbone) [6]. The baselines are based on either English or Spanish BERT models, depending on the language. The task’s github repository contains the code of the baselines.

Official Ranking

TASK 1 - ENGLISH
POSITION	TEAM	MCC	F1-MACRO	F1-CONSPIRACY	F1-CRITICAL
1	IUCL	0.8388	0.9194	0.8947	0.9441
2	AI_Fusion	0.8303	0.9147	0.8866	0.9429
3	SINAI	0.8297	0.9149	0.8886	0.9412
4	ezio	0.8212	0.9097	0.8792	0.9402
5	hinlole	0.8198	0.9098	0.8811	0.9386
6	Zleon	0.8195	0.9096	0.8804	0.9388
7	virmel	0.8192	0.9092	0.8793	0.9391
8	inaki	0.8149	0.9072	0.8770	0.9374
9	yeste	0.8124	0.9057	0.8746	0.9368
10	auxR	0.8088	0.9043	0.8739	0.9347
11	Elias&Sergio	0.8034	0.9012	0.8687	0.9338
12	theateam	0.8031	0.8999	0.8650	0.9347
13	trustno1	0.7983	0.8991	0.8675	0.9307
14	DSVS	0.7970	0.8985	0.8674	0.9296
15	sail	0.7969	0.8978	0.8687	0.9268
16	ojo-bes	0.7969	0.8981	0.8648	0.9314
17	RD-IA-FUN	0.7965	0.8977	0.8636	0.9317
	baseline-BERT	0.7964	0.8975	0.8632	0.9318
18	aish_team	0.7917	0.8944	0.8580	0.9309
19	rfenthusiasts	0.7902	0.8948	0.8605	0.9291
20	Dap_upv	0.7898	0.8944	0.8593	0.9294
21	oppositional_opposition	0.7894	0.8935	0.8571	0.9300
22	RD-IA-FUN	0.7894	0.8947	0.8617	0.9276
23	miqarn	0.7881	0.8938	0.8593	0.9283
24	CHEEXIST	0.7875	0.8932	0.8576	0.9287
25	tulbure	0.7872	0.8917	0.8536	0.9297
26	XplaiNLP	0.7871	0.8922	0.8550	0.9294
27	TheGymNerds	0.7854	0.8923	0.8567	0.9278
28	nlpln	0.7844	0.8922	0.8580	0.9263
29	RalloRico	0.7771	0.8879	0.8559	0.9198
30	LasGarcias	0.7758	0.8855	0.8447	0.9263
31	zhengqiaozeng	0.7758	0.8866	0.8476	0.9256
32	ALC_UPV_JD_2	0.7725	0.8860	0.8491	0.9230
33	LorenaEloy	0.7713	0.8847	0.8455	0.9239
34	lnr-alhu	0.7708	0.8853	0.8488	0.9219
35	NACKO	0.7692	0.8838	0.8446	0.9230
36	paranoia-pulverizers	0.7680	0.8838	0.8462	0.9215
37	DiTana	0.7653	0.8806	0.8490	0.9123
38	FredYNed	0.7643	0.8806	0.8392	0.9220
39	dannuchihaxxx	0.7643	0.8801	0.8377	0.9224
40	lnr-detectives	0.7631	0.8806	0.8472	0.9141
41	TargaMarhuenda	0.7617	0.8807	0.8424	0.9190
42	Trainers	0.7596	0.8797	0.8412	0.9182
43	thetaylorswiftteam	0.7577	0.8755	0.8302	0.9208
44	locasporlnr	0.7575	0.8787	0.8399	0.9174
45	lnr-adri	0.7552	0.8759	0.8326	0.9192
46	TokoAI	0.7542	0.8767	0.8363	0.9172
47	ede	0.7539	0.8769	0.8384	0.9155
48	lnr-verdnav	0.7529	0.8746	0.8308	0.9185
49	lnr-dahe	0.7488	0.8736	0.8308	0.9163
50	epistemologos	0.7486	0.8742	0.8341	0.9143
51	lucia&ainhoa	0.7473	0.8733	0.8316	0.9150
52	pistacchio	0.7414	0.8678	0.8200	0.9155
53	lnr-BraulioPaula	0.7393	0.8658	0.8165	0.9152
54	Marc_Coral	0.7392	0.8663	0.8176	0.9150
55	Ramon&Cajal	0.7284	0.8633	0.8169	0.9096
56	lnr-lladrogal	0.7253	0.8603	0.8106	0.9100
57	lnr-fanny-nuria	0.7253	0.8594	0.8082	0.9106
58	MarcosJavi	0.7190	0.8583	0.8097	0.9069
59	lnr-cla	0.7168	0.8573	0.8085	0.9061
60	lnr-jacobantonio	0.7168	0.8573	0.8085	0.9061
61	MUCS	0.7162	0.8538	0.7994	0.9082
62	lnr-aina-julia	0.7157	0.8574	0.8102	0.9046
63	LaDolceVita	0.7072	0.8519	0.8000	0.9037
64	alopfer	0.7056	0.8518	0.8012	0.9023
65	lnr-luqrud	0.7056	0.8518	0.8012	0.9023
66	LNR-JoanPau	0.7051	0.8426	0.7793	0.9058
67	lnr-carla	0.7000	0.8476	0.7932	0.9020
68	lnr-Inetum	0.6981	0.8328	0.7617	0.9039
69	lnr-antonio	0.6852	0.8300	0.7598	0.9002
70	LluisJorge	0.6784	0.8382	0.7830	0.8934
71	anselmo-team	0.6725	0.8341	0.7752	0.8930
72	lnr-pavid	0.5959	0.7974	0.7297	0.8651
73	LNRMADME	0.5469	0.7717	0.6914	0.8521
74	lnr-mariagb_elenaog	0.5069	0.7250	0.5966	0.8534
75	LNR_08	0.4429	0.6834	0.5276	0.8391
76	Kaprov	0.3700	0.6240	0.4224	0.8255
77	lnr_cebusqui	0.0482	0.4760	0.1847	0.7674
78	jtommor	0.0403	0.5167	0.3312	0.7023
79	eledu	-0.4598	0.2350	0.2740	0.1960
80	david-canet	-0.6310	0.1632	0.1883	0.1381
81	lnr-guilty	-0.6595	0.1433	0.2247	0.0619
82	lnrANRI	-0.7551	0.1072	0.1474	0.0670
83	ROCurve	-0.8009	0.0884	0.1112	0.0656

TASK 1 - SPANISH
POSITION	TEAM	MCC	F1-MACRO	F1-CONSPIRACY	F1-CRITICAL
1	SINAI	0.7429	0.8705	0.8319	0.9091
2	auxR	0.7205	0.8572	0.8112	0.9032
3	RD-IA-FUN	0.7028	0.8497	0.8035	0.8960
4	Elias&Sergio	0.6971	0.8485	0.8087	0.8884
5	AI_Fusion	0.6872	0.8419	0.7931	0.8908
6	zhengqiaozeng	0.6871	0.8417	0.7925	0.8909
7	virmel	0.6854	0.8426	0.8022	0.8831
8	trustno1	0.6848	0.8400	0.7895	0.8906
9	Zleon	0.6826	0.8410	0.7955	0.8865
10	ojo-bes	0.6817	0.8395	0.8026	0.8764
11	tulbure	0.6722	0.8293	0.7699	0.8887
12	sail	0.6719	0.8299	0.7713	0.8884
13	nlpln	0.6681	0.8339	0.7872	0.8806
	baseline-BERT	0.6681	0.8339	0.7872	0.8806
14	pistacchio	0.6678	0.8327	0.7822	0.8833
15	rfenthusiasts	0.6656	0.8255	0.7643	0.8868
16	XplaiNLP	0.6622	0.8274	0.7708	0.8840
17	yeste	0.6609	0.8291	0.7770	0.8812
18	oppositional_opposition	0.6601	0.8274	0.7724	0.8825
19	epistemologos	0.6562	0.8264	0.7728	0.8801
20	miqarn	0.6562	0.8264	0.7728	0.8801
21	theateam	0.6557	0.8252	0.7695	0.8810
22	ezio	0.6535	0.8242	0.7683	0.8801
23	lucia&ainhoa	0.6524	0.8260	0.7765	0.8754
24	TargaMarhuenda	0.6516	0.8240	0.7692	0.8787
25	TokoAI	0.6516	0.8240	0.7692	0.8787
26	paranoia-pulverizers	0.6494	0.8246	0.7762	0.8730
27	NACKO	0.6467	0.8232	0.7739	0.8726
28	ALC_UPV_JD_2	0.6467	0.8227	0.7705	0.8748
29	DSVS	0.6462	0.8231	0.7753	0.8709
30	RD-IA-FUN	0.6445	0.8160	0.7523	0.8796
31	locasporlnr	0.6437	0.8216	0.7709	0.8723
32	DiTana	0.6377	0.8187	0.7677	0.8696
33	lnr-BraulioPaula	0.6358	0.8173	0.7731	0.8615
34	Dap_upv	0.6306	0.8115	0.7493	0.8737
35	TheGymNerds	0.6306	0.8106	0.7470	0.8743
36	MUCS	0.6293	0.8060	0.7363	0.8756
37	LasGarcias	0.6247	0.8122	0.7594	0.8649
38	lnr-dahe	0.6196	0.8066	0.7437	0.8694
39	lnr-adri	0.6194	0.8060	0.7422	0.8698
40	hinlole	0.6192	0.8048	0.7391	0.8706
41	RalloRico	0.6105	0.8018	0.7370	0.8666
42	lnr-aina-julia	0.6103	0.7978	0.7264	0.8692
43	lnr-verdnav	0.6101	0.7991	0.7298	0.8684
44	thetaylorswiftteam	0.6066	0.8025	0.7436	0.8613
45	lnr-alhu	0.6024	0.7991	0.7358	0.8624
46	lnr-luqrud	0.6010	0.7945	0.7237	0.8654
47	lnr-lladrogal	0.5967	0.7942	0.7256	0.8627
48	ede	0.5965	0.7967	0.7341	0.8593
49	Fred&Ned	0.5931	0.7940	0.7283	0.8597
50	LaDolceVita	0.5921	0.7818	0.6981	0.8656
51	LNR-JoanPau	0.5920	0.7916	0.7218	0.8614
52	anselmo-team	0.5899	0.7860	0.7085	0.8634
53	Ramon&Cajal	0.5858	0.7916	0.7281	0.8552
54	lnr-fanny-nuria	0.5813	0.7874	0.7181	0.8567
55	lnr-antonio	0.5736	0.7816	0.7071	0.8561
56	LluisJorge	0.5690	0.7750	0.6929	0.8571
57	lnr-cla	0.5651	0.7788	0.7055	0.8520
58	lnr-jacobantonio	0.5651	0.7788	0.7055	0.8520
59	lnr-pavid	0.5569	0.7771	0.7089	0.8453
60	alopfer	0.5520	0.7727	0.6984	0.8470
61	LNRMADME	0.5490	0.7704	0.6937	0.8471
62	lnr-carla	0.5484	0.7686	0.6890	0.8482
63	LorenaEloy	0.5433	0.7621	0.6751	0.8492
64	CHEEXIST	0.5379	0.5995	0.5621	0.5456
65	lnr-guilty	0.5273	0.7620	0.6880	0.8360
66	eledu	0.5057	0.7263	0.6098	0.8429
67	lnr-mariagb_elenaog	0.4966	0.7325	0.6271	0.8379
68	dannuchihaxxx	0.4727	0.7310	0.6382	0.8238
69	lnr-detectives	0.4029	0.6734	0.6509	0.6960
70	LNR_08	0.0608	0.4771	0.2000	0.7542
71	jtommor	0.0105	0.5051	0.3813	0.6288
72	lnr-Inetum	0.0000	0.3880	0.0000	0.7760
73	Marc_Coral	0.0000	0.2679	0.5359	0.0000
74	MarcosJavi	-0.0389	0.3887	0.0054	0.7720
75	lnr_cebusqui	-0.4112	0.2481	0.3466	0.1496
76	david-canet	-0.5058	0.2114	0.3029	0.1199
77	lnrANRI	-0.6146	0.1766	0.1939	0.1593
78	ROCurve	-0.6457	0.1628	0.1770	0.1485

TASK 2 - ENGLISH
POSITION	TEAM	span-F1	span-P	span-R	micro-span-F1
1	tulbure	0.6279	0.5859	0.6790	0.6120
2	Zleon	0.6089	0.5537	0.6881	0.5856
3	hinlole	0.5886	0.5243	0.6834	0.5571
4	oppositional_opposition	0.5866	0.5347	0.6586	0.5344
5	AI_Fusion	0.5805	0.5585	0.6082	0.5437
6	virmel	0.5742	0.5235	0.6477	0.5540
7	miqarn	0.5739	0.5184	0.6462	0.5325
8	TargaMarhuenda	0.5701	0.5161	0.6477	0.5437
9	ezio	0.5694	0.5229	0.6340	0.5389
10	zhengqiaozeng	0.5666	0.5122	0.6485	0.5421
11	Elias&Sergio	0.5627	0.5149	0.6364	0.5248
12	DSVS	0.5598	0.5332	0.6012	0.5287
13	CHEEXIST	0.5524	0.4767	0.6845	0.5299
14	rfenthusiasts	0.5479	0.5381	0.5666	0.5408
15	ALC_UPV_JD_2	0.5377	0.4643	0.6562	0.4956
	baseline-BETO	0.5323	0.4684	0.6334	0.4998
16	Dap_upv	0.5272	0.4617	0.6297	0.4973
17	aish_team	0.5213	0.4181	0.7456	0.2571
18	SINAI	0.4582	0.5553	0.4279	0.4571
19	Trainers	0.3382	0.5124	0.2609	0.2858
20	nlpln	0.3339	0.5286	0.3303	0.2710
21	ROCurve	0.2996	0.3154	0.3031	0.3425
22	TokoAI	0.2760	0.1870	0.6119	0.2677
23	DiTana	0.2756	0.5259	0.1947	0.2599
24	TheGymNerds	0.2070	0.2076	0.2127	0.2329
25	epistemologos	0.1709	0.1286	0.3244	0.1201
26	theateam	0.1503	0.1401	0.1652	0.0387
27	LaDolceVita	0.0726	0.2040	0.0453	0.0630
28	kaprov	0.0150	0.0261	0.0165	0.0600

TASK 2 - SPANISH
POSITION	TEAM	span-F1	span-P	span-R	micro-span-F1
1	tulbure	0.6129	0.6159	0.6129	0.6108
2	Zleon	0.5875	0.5439	0.6474	0.5939
3	AI_Fusion	0.5777	0.5437	0.6189	0.5843
4	CHEEXIST	0.5621	0.5379	0.5995	0.5456
5	virmel	0.5616	0.4963	0.6584	0.5620
6	miqarn	0.5603	0.5117	0.6273	0.5618
7	DSVS	0.5529	0.5384	0.5785	0.5323
8	TargaMarhuenda	0.5364	0.5128	0.5710	0.5385
9	Elias&Sergio	0.5151	0.4864	0.5533	0.5231
10	hinlole	0.4994	0.4530	0.5740	0.4890
	baseline-BETO	0.4934	0.4533	0.5621	0.4952
11	Dap_upv	0.4914	0.4555	0.5474	0.4917
12	zhengqiaozeng	0.4903	0.4507	0.5494	0.4874
13	ALC_UPV_JD_2	0.4885	0.4509	0.5458	0.4683
14	ezio	0.4869	0.4623	0.5229	0.4947
15	nlpln	0.4672	0.5174	0.4426	0.2961
16	rfenthusiasts	0.4666	0.5104	0.4341	0.4697
17	SIANI	0.4151	0.4630	0.4054	0.4781
18	TheGymNerds	0.3984	0.3621	0.4483	0.5024
19	DiTana	0.3004	0.4490	0.2362	0.3117
20	ROCurve	0.2649	0.2706	0.2627	0.3562
21	TokoAI	0.1878	0.1189	0.5659	0.1739
22	epistemologos	0.1657	0.1906	0.1864	0.1534
23	LaDolceVita	0.1056	0.1158	0.0975	0.1321
24	theateam	0.0994	0.1051	0.0962	0.0358
25	oppositional_opposition	0.0037	0.0349	0.0022	0.0014

Related Work

• [1] K. M. Douglas and R. M. Sutton, “What Are Conspiracy Theories? A Definitional Approach to Their Correlates, Consequences, and Communication,” Annu. Rev. Psychol., vol. 74, no. 1, Jan. 2023.
• [2] A. Giachanou, B. Ghanem, and P. Rosso, “Detection of conspiracy propagators using psycho-linguistic characteristics,” Journal of Information Science, vol. 49, no. 1, pp. 3–17, Feb. 2023.
• [3] Chicco, Davide, Tötsch, Niklas, and Jurman, Giuseppe, “The Matthews correlation coefficient (MCC) is more reliable than balanced accuracy, bookmaker informedness, and markedness in two-class confusion matrix evaluation,” BioData Mining, vol. 14, no. 1, pp. 13, Feb. 04, 2021.
• [4] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.” arXiv, May 24, 2019.
• [5] Da San Martino, Giovanni, Yu, Seunghak, Barrón-Cedeño, Alberto, Petrov, Rostislav, and Nakov, Preslav, “Fine-Grained Analysis of Propaganda in News Articles,” in Proceedings of the EMNLP-IJCNLP 2019, Hong Kong, China, pp. 5636–5646, Association for Computational Linguistics, Nov. 2019.
• [6] S. Ruder, “An Overview of Multi-Task Learning in Deep Neural Networks.” arXiv, Jun. 15, 2017.
• [7] J. D. Moffitt, C. King, and K. M. Carley, “Hunting Conspiracy Theories During the COVID-19 Pandemic,” Social Media + Society, vol. 7, no. 3, p. 20563051211043212, Jul. 2021.
• [8] K. Pogorelov, D. T. Schroeder, S. Brenner, and J. Langguth, “FakeNews: Corona Virus and Conspiracies Multimedia Analysis Task at MediaEval 2021,” p. 3.
• [9] K. Pogorelov, D. T. Schroeder, S. Brenner, A. Maulana, and J. Langguth, “Combining Tweets and Connections Graph for FakeNews Detection at MediaEval 2022”.
• [10] Y. Peskine, G. Alfarano, I. Harrando, P. Papotti, and R. Troncy, “Detecting COVID-19-Related Conspiracy Theories in Tweets,” p. 3.
• [11] Y. Peskine, P. Papotti, and R. Troncy, “Detection of COVID-19-Related Conpiracy Theories in Tweets using Transformer-Based Models and Node Embedding Techniques”.
• [12] D. Korenčić, I. Grubišić, A. H. Toselli, B. Chulvi, and P. Rosso, “Tackling Covid-19 Conspiracies on Twitter using BERT Ensembles, GPT-3 Augmentation, and Graph NNs”.
• [13] Y. Peskine, D. Korenčić, I. Grubisic, P. Papotti, R. Troncy, and P. Rosso, “Definitions Matter: Guiding GPT for Multi-label Classification,” in Findings of the Association for Computational Linguistics: EMNLP 2023, Singapore: Association for Computational Linguistics, 2023, pp. 4054–4063.
• [14] E. Funkhouser, “A tribal mind: Beliefs that signal group identity or commitment,” Mind & Language, vol. 37, no. 3, pp. 444–464, 2022.
• [15] B. Franks, A. Bangerter, M. W. Bauer, M. Hall, and M. C. Noort, “Beyond ‘Monologicality’? Exploring Conspiracist Worldviews,” Frontiers in Psychology, vol. 8, 2017.
• [16] S. Phadke, M. Samory, and T. Mitra, “What Makes People Join Conspiracy Communities? Role of Social Factors in Conspiracy Engagement,” Proc. ACM Hum.-Comput. Interact., vol. 4, no. CSCW3, p. 223:1-223:30, Jan. 2021.
• [17] R. Böhm, H. Rusch, and J. Baron, “The psychology of intergroup conflict: A review of theories and measures,” Journal of Economic Behavior & Organization, vol. 178, pp. 947–962, Oct. 2020.
• [18] P. Wagner-Egger, A. Bangerter, S. Delouvée, and S. Dieguez, “Awake together: Sociopsychological processes of engagement in conspiracist communities,” Current Opinion in Psychology, vol. 47, p. 101417, Oct. 2022.

Important Dates

• February 23, 2024: Train data release
• May 30, 2024: Software submission deadline
• June 15, 2024: Participant paper submission Midnight CEST
• July 1st, 2024: Peer review notification
• July 7th, 2024: Camera-ready participant papers submission Midnight CEST

Funds

Task Committee