The research carried out at PAN's shared tasks informs the development of new digital text forensics technology. For reproducibility sake, the prototypes submitted for evaluation are made available by participants open source, as executables on TIRA, or both. The choice of license is at the discretion of participatns, who retain copyright of their software.
At PAN, it is our goal to carry out reproducible research. While the evaluation resources created for our shared tasks benefit future evaluations, even shared tasks lack reproducibility. Therefore, we ask participants to submit working executables of their software. To facilitate the submission of software, we employ TIRA (tira.io), a platform that implements the evaluation as a service paradigm.
To get started, send us an email with your preferred choice of operating system. We offer Ubuntu Linux 14 LTS (Desktop or Server) as well as Windows 7. We will set up a virtual machine (VM) for you and pass you the access credentials. You can access your VM via SSH and remote desktop. Once you have access, please install your software inside the virtual machine.
Reproducibility is one of the cornerstones of science; yet, in recent years it was shown that many scientific contributions are irreproducible. For shared tasks in particular, the software developed by participants often go missing soon after the event has passed. Moreover, the test datasets used in shared tasks are released to participants, creating an evaluation bias. All of these problems are solved by submitting software instead of their run output.
To work with TIRA, your software has to be executable from the command line and it must accept the following parameters: