Synopsis

  • Task: Given a news article text, decide whether it follows a hyperpartisan argumentation, i.e., whether it exhibits blind, prejudiced, or unreasoning allegiance to one party, faction, cause, or person.
  • Input: [data]
  • Evaluation: [code]
  • Submission: [submit]
  • Baseline: [code]

Video

Data

The competition is over, but you can still request access to the data.

Evaluation

Main performance measure is accuracy on a balanced set of articles. In addition, we measure precision, recall, and F1-score for the hyperpartisan class. Please find the SemEval leaderboard here. However, note that you can still submit your approaches (see Ongoing Submissions).

Ongoing Submissions

We continue to allow submissions. New submissions will appear on the TIRA leaderboard [labels-by-article, labels-by-publisher]. When you publish your code on Github, we will add it to our list. To participate, please send a mail to the organizer's mailing list with your choice of a operating system (Ubuntu/Windows) and code name and we will provide you with a virtual machine. You can find help here and on the PAN mailing list.

Your software must be executable from the command line and not require Internet access during the evaluation. Note that you retain full copyrights of your software, but agree to grant us usage rights only for the purpose of the competition.

We provide a random baseline to illustrate the output of a submission and a term frequency extractor to illustrate how to read the dataset. For features, see the code from our ACL'18 publication for inspiration.

Grand Prize

In order to encourage developers to share their software so that everyone can profit from their work, we announced a grand prize of $1,000 to the best-performing submission that had its code published open source before the SemEval conference (which has been held at NAACL-HLT on June 6-7 in Minneapolis, USA).

The grand prize has been won by team bertha-von-suttner (Ye Jiang, Johann Petrak, Xingyi Song, Kalina Bontcheva, Diana Maynard from the University of Sheffield, United Kingdom). Well done!

Xingyi Song, Ye Jiang, and Diana Maynard (from left to right, all University of Sheffield, United Kingdom) of the winning team bertha-von-suttner with the certificate of the grand prize.

Results

Main Leaderboard

Main leaderboard for the SemEval 2019 Task 4 on Hyperpartisan News Detection, based on the unpublished test set for labels-by-article.
#TeamMembersAccuracyPrecisionRecallF1Code
1bertha-von-suttnerYe Jiang, Johann Petrak, Xingyi Song, Kalina Bontcheva, Diana Maynard (University of Sheffield, United Kingdom)0.8220.8710.7550.809
2vernon-fenwickVertika Srivastava, Yeon Hyang Kim, Divya Prakash, Ankita Gupta, Rohit R.R, Sudeep Kumar Sahoo (Samsung R&D Institute India-Bangalore)0.8200.8150.8280.821
3sally-smedleyShota Sasaki1, Kazuaki Hanawa1, Tsubasa Tagami1, Hiroki Ouchi1,2, Jun Suzuki1,2, Kentaro Inui1,2 (1Tohoku University / 2RIKEN AIP, Japan)0.8090.8230.7870.805
4tom-jumbo-grumboChia-Lun Yeh, Nava Tintarev (TU Delft, Netherlands), Babak Loni, Anne Schuth (De Persgroep)0.8060.8580.7320.790
5dick-prestonTim Isbister (FOI, Sweden)0.8030.7930.8180.806
6borat-sagdiyevNiko Palić, Juraj Vladika, Dominik Čubelić, Ivan Lovrenčić, Jan Šnajder, Maja Buljan (TakeLab, University of Zagreb, Croatia)0.7910.8830.6720.763
7morboMaja Karasalo, Andreas Horndahl. Magnus Rosell, Fredrik Johansson (FOI, Sweden)0.7900.7720.8220.796
8howard-bealeOzan Arkan Can, Osman Mutlu (Koç University, Turkey), Erenay Dayanık (University of Stuttgart, Germany)0.7830.8370.7040.765
9ned-leedsBozhidar Stevanoski, Sonja Gievska (Ss. Cyril and Methodius Skopje, Macedonia)0.7750.8650.6530.744
10clint-buchananPedro Sandoval Segura, Mehdi Drissi, Vivaswat Ojha, Julie Medero (Harvey Mudd College, United States)0.7710.8320.6780.747
11yeon-ziNayeon Lee, Zihan Liu, Pascale Fung (HKUST, Hong Kong)0.7580.7440.7870.765
12tony-vincenzoTodor Staykovski (Bulgaria)0.7500.7640.7230.743
13paparazzoDuc-Vu Nguyen, Dang Van Thin, Ngan Luu-Thuy Nguyen (University of Information Technology - Vietnam National University Ho Chi Minh City, Vietnam)0.7470.7540.7320.743
14steve-martinYoungjun Joo (Yonsei University, South Korea)0.7450.8530.5920.699
15eddie-brockAntonio Šajatović, David Lozić, Doria Šarić, Filip Boltužić, Jan Šnajder (TakeLab, University of Zagreb, Croatia)0.7440.7820.6750.725
16ankh-morpork-timesCarla Perez-Almendros, Luis Espinosa-Anke, Steven Schockaert (Cardiff University, United Kingdom)0.7420.8110.6310.710
17spider-jerusalemAmal Alabdulkarim, Tariq Alhindi (Columbia Univerity, United States)0.7420.8140.6270.709
18carl-kolchakCeline Park, Celena Chen, Jason Dwyer, Julie Medero (Harvey Mudd College, United States)0.7390.7290.7610.745
19doris-martinRodrigo Agerri (IXA NLP Group, University of the Basque Country UPV/EHU)0.7370.7540.7040.728
20pistachonAbdul Saleh1, Alberto Barrón-Cedeño2, Ramy Baly1, Giovanni Da San Martino2, Mitra Mohtarami1, Preslav Nakov2 (1MIT, USA / 2QCRI, Qatar)0.7290.7240.7420.733
21joseph-rouletabilleJose G Moreno, Gilles Hubert, Karen Pinel-Sauvagnat, Yoann Pitarch (IRIT - Université de Toulouse, France)0.7250.7880.6150.691
22fernando-pessaAndré Cruz, Gil Rocha, Rui Sousa Silva, Henrique Lopes Cardoso (LIACC and DEI/FEUP, Portugal)0.7170.8060.5700.668
23pioquinto-manterolaTed Pedersen, Saptarshi Sengupta (University of Minnesota, United States)0.7040.7410.6270.679
24miles-clarksonChiyu Zhang, Arun Rajendran, Muhammad Abdul-Mageed (The University of British Columbia, Canada)0.6830.7450.5570.638
25xenophilius-lovegoodAlbin Zehe, Andreas Hotho, Lena Hettinger, Christian Hauptmann, Stefan Ernst (University of Würzburg, Germany)0.6750.6190.9140.738
26orwellian-timesJürgen Knauth (University of Göttingen, Germany)0.6720.6540.7290.690
27tintinYves Bestgen (CECL - UCLouvain, Belgium)0.6560.6420.7070.673
28d-x-beaumontJake Palanker, Evan Amason, Mary Clare Shen, Julie Medero (Harvey Mudd College, United States)0.6530.5970.9390.730
29jack-ryderDaniel Shaprin (Sofia University, Bulgaria)0.6460.6460.6460.646
30kermit-the-frogTalita Rani Anthonio (University of Groningen, Netherlands), Lennart Albertus Mathijs Kloppenburg (Netherlands)0.6210.5820.8600.694
31billy-batsonTim Kreutz, Stiene Praet, Jeroen Peeters, Walter Daelemans (University of Antwerp, Belgium)0.6150.5680.9620.714
32peter-brinkmannMichael Färber (University of Freiburg, Germany), Agon Qurdina (University of Prishtina, Kosovo), Lule Ahmedi (University of Prishtina, Kosovo)0.6020.5600.9550.706
33anson-brysonCordelia Stiff, Julie Medero (Harvey Mudd College, United States)0.5920.7200.3030.426
34sarah-jane-smithVijayasaradhi Indurthi (IIIT Hyderabad, India)0.5910.5540.9330.695
35kit-kittredgeRebekah Cramerus, Tatjana Scheffler (University of Potsdam, Germany)0.5780.5470.9080.683
36brenda-starrOlga Papadopoulou, Markos Zampoglou, Symeon Papadopoulos, Ioannis Kompatsiaris (ITI-CERTH, Information Technologies Institute, Centre for Research and Technology Hellas, Greece)0.5750.5420.9710.696
37harry-fribergPeter Sumbler, Nazanin Afsarmanesh, Nina Viereckel, Jussi Karlgren (Gavagai AB, Sweden)0.5650.5370.9490.686
38robin-scherbatskyMaarten Marx, Alexandra Akut (University of Amsterdam, Netherlands)0.5510.5420.6620.596
39clark-kentRamneek Kaur, Baani Leen Kaur Jolly, Viresh Gupta, Tanmoy Chakraborty (IIIT-Delhi, India)0.5480.6830.1780.283
40murphy-brownAnamika Sen, Jiepu Jiang (Virginia Tech, United States)0.5290.5180.8220.635
41peter-parkerYuanzhen Lin, Zhiyuan Ning, Ruichao Zhong, Weifeng Su, Jeffson Fong (Beijing Normal University - Hong Kong Baptist University United International College, China)0.5030.5020.7710.608
42john-kingSrijan Bansal, Arsalan Saad, Shayan Shafquat, Nitin Choudhary, Jasabanta Patro (India)0.4620.4600.4430.451

Labels-by-publisher Leaderboard

Secondary leaderboard for the SemEval 2019 Task 4 on Hyperpartisan News Detection, based on the unpublished test set for labels-by-publisher.
#TeamMembersAccuracyPrecisionRecallF1Code
1tintinYves Bestgen (CECL - UCLouvain, Belgium)0.7060.7420.6320.683
2joseph-rouletabilleJose G Moreno, Gilles Hubert, Karen Pinel-Sauvagnat, Yoann Pitarch (IRIT - Université de Toulouse, France)0.6800.6400.8270.721
3brenda-starrOlga Papadopoulou, Markos Zampoglou, Symeon Papadopoulos, Ioannis Kompatsiaris (ITI-CERTH, Information Technologies Institute, Centre for Research and Technology Hellas, Greece)0.6640.6270.8070.706
4xenophilius-lovegoodAlbin Zehe, Andreas Hotho, Lena Hettinger, Christian Hauptmann, Stefan Ernst (University of Würzburg, Germany)0.6630.6320.7810.699
5yeon-ziNayeon Lee, Zihan Liu, Pascale Fung (HKUST, Hong Kong)0.6630.6350.7660.694
6miles-clarksonChiyu Zhang, Arun Rajendran, Muhammad Abdul-Mageed (The University of British Columbia, Canada)0.6520.6120.8320.705
7jack-ryderDaniel Shaprin (Sofia University, Bulgaria)0.6450.6000.8690.710
8bertha-von-suttnerYe Jiang, Johann Petrak, Xingyi Song, Kalina Bontcheva, Diana Maynard (University of Sheffield, United Kingdom)0.6430.6160.7620.681
9howard-bealeOzan Arkan Can, Osman Mutlu (Koç University, Turkey), Erenay Dayanık (University of Stuttgart, Germany)0.6410.6060.8060.692
10eddie-brockAntonio Šajatović, David Lozić, Doria Šarić, Filip Boltužić, Jan Šnajder (TakeLab, University of Zagreb, Croatia)0.6310.6810.4910.571
11sally-smedleyShota Sasaki1, Kazuaki Hanawa1, Tsubasa Tagami1, Hiroki Ouchi1,2, Jun Suzuki1,2, Kentaro Inui1,2 (1Tohoku University / 2RIKEN AIP, Japan)0.6250.6400.5710.603
12murphy-brownAnamika Sen, Jiepu Jiang (Virginia Tech, United States)0.6230.6150.6590.636
13tom-jumbo-grumboChia-Lun Yeh, Nava Tintarev (TU Delft, Netherlands), Babak Loni, Anne Schuth (De Persgroep)0.6190.5920.7620.667
14sarah-jane-smithVijayasaradhi Indurthi (IIIT Hyderabad, India)0.6120.5860.7650.664
15pistachonAbdul Saleh1, Alberto Barrón-Cedeño2, Ramy Baly1, Giovanni Da San Martino2, Mitra Mohtarami1, Preslav Nakov2 (1MIT, USA / 2QCRI, Qatar)0.6080.6380.4990.560
16morboMaja Karasalo, Andreas Horndahl. Magnus Rosell, Fredrik Johansson (FOI, Sweden)0.6010.5870.6790.630
17fernando-pessaAndré Cruz, Gil Rocha, Rui Sousa Silva, Henrique Lopes Cardoso (LIACC and DEI/FEUP, Portugal)0.6000.5850.6810.630
18steve-martinYoungjun Joo (Yonsei University, South Korea)0.5970.6250.4830.545
19borat-sagdiyevNiko Palić, Juraj Vladika, Dominik Čubelić, Ivan Lovrenčić, Jan Šnajder, Maja Buljan (TakeLab, University of Zagreb, Croatia)0.5920.6440.4120.502
20kermit-the-frogTalita Rani Anthonio (University of Groningen, Netherlands), Lennart Albertus Mathijs Kloppenburg (Netherlands)0.5890.5750.6810.623
21ankh-morpork-timesCarla Perez-Almendros, Luis Espinosa-Anke, Steven Schockaert (Cardiff University, United Kingdom)0.5880.6460.3890.486
22ned-leedsBozhidar Stevanoski, Sonja Gievska (Ss. Cyril and Methodius Skopje, Macedonia)0.5730.5460.8570.667
23orwellian-timesJürgen Knauth (University of Göttingen, Germany)0.5370.5300.6580.587
24paparazzoDuc-Vu Nguyen, Dang Van Thin, Ngan Luu-Thuy Nguyen (University of Information Technology - Vietnam National University Ho Chi Minh City, Vietnam)0.5300.5300.5410.535
25robin-scherbatskyMaarten Marx, Alexandra Akut (University of Amsterdam, Netherlands)0.5240.8220.0620.116
26clark-kentRamneek Kaur, Baani Leen Kaur Jolly, Viresh Gupta, Tanmoy Chakraborty (IIIT-Delhi, India)0.5190.5650.1700.261
27dick-prestonTim Isbister (FOI, Sweden)0.5140.5200.3520.420
28peter-brinkmannMichael Färber (University of Freiburg, Germany), Agon Qurdina (University of Prishtina, Kosovo), Lule Ahmedi (University of Prishtina, Kosovo)0.4970.4960.3440.406

Task Committee