Given a news article text, decide whether it follows a hyperpartisan argumentation, i.e., whether it exhibits blind, prejudiced, or unreasoning allegiance to one party, faction, cause, or person.
Upon registration, we provide a two-part dataset of articles for training your algorithm (may take up to 24 hours). We continuously clean the dataset (also based on your feedback) and will send you new versions as they come out.
Register yourself or your team here. This task will use the PAN mailing list for communication, so please sign up there to ask questions to us or other participants. If you have specific requests, please write to the organizer's mailing list.
When you registered, you will get remote access to a virtual machine (Windows or Linux) to deploy the task software in. Your software must be executable from the command line and not require Internet access during the evaluation period.
Note that you retain full copyrights of your software, but agree to grant us usage rights only for the purpose of the competition.
We provide a random baseline to illustrate the output of a submission and a term frequency extractor to illustrate how to read the dataset. For features, see the code from our ACL'18 publication for inspiration.
Evaluation (13 Dec 2018, 23 Jan 2019 (both AoE))
You will self-evaluate your software using TIRA. Main performance measure will be accuracy on a balanced set of articles. In addition, we will measure precision, recall, and F1-score for the hyperpartisan class (evaluation script).
You will be able to designate up to three test-data runs for this competition: one submitted before 13 Dec ("Early Bird") and two more submitted before 23 Jan.
After evaluation, participants are required to submit and review task description papers following the SemEval timeline.
Grand Prize (Jun 2019)
We want to encourage developers to share their software so that everyone can profit from their work. We are thus excited to announce a grand prize of $1,000 to the best-performing submission that ha its code published open source before the SemEval conference (which is held at NAACL-HLT on June 6-7 in Minneapolis, USA).