One of the most remarkable aspects of Large Language Models is their ease of writing creative texts; under certain conditions, they have been shown to match and improve average human writing skills. But is their writing truly creative, or just a repetition of the clichés they have been pre-trained with? Do they have a distinctive creative writing style? What is the role of (human) prompting in the creation process?
In the talk, I will discuss the creative writing potential of LLMs and their intrinsic limitations, paying special attention to the experiments carried out at UNED. These include a contest between GPT-4 and one of the best contemporary novelists in Spanish, Patricio Pron. The contest is inspired by past AI duels (such as DeepBlue vs Kasparov and AlphaGo vs Lee Sidol), and was designed to test whether LLMs can already challenge a top (rather than an average) fiction writer.
We use TIRA for all submissions to PAN. Please go to tira.io, create an account, and register for the individual tasks you want to participate in.
You need to submit your software or your results via TIRA.
You can find all submission guides in TIRA's forum.
Data
You can download PAN's datasets here. For details, please check the individual task's website.
Evaluation and Baseline Code
All code used at PAN is published at GitHub.
You can find all validators, evaluators, and baselines there.
Software Submissions
PAN promotes reproducible science with software submissions.
Please prepare and submit your software as Docker image(s).
You can find guides and examples in the resources linked above.
Some tasks allow only software submissions and only release the test data after the conference.
Paper Submission and Presentation
PAN is co-located with CLEF 2024 in Grenoble.
Every participant is expected to write a notebook paper describing their approach to CLEF (published at CEUR-WS, which is indexed by DBLP).
At the CLEF conference, all submissions will be presented as talks or posters. CLEF will be a hybrid conference.
