Should we allow questions asking for identification of the outputs of language models?

Question

A user new to the site recently asked for the identification of a poem output by a large language model (some version of OpenAI's ChatGPT). The user had asked the model for a poem in German on a given subject, and the model had output some lines of German verse, together with an attribution to the (real) poet Barbara Köhler. The output of the model was so convincing that the user was prepared to entertain the hypothesis that this might be a real poem of Köhler's.

The question has now been closed, but in order to have a policy that we can refer to if something similar happens in future, I'm posting this question here: should we allow questions asking for the identification of texts that have been output by large language models?

Answer 1

We should disallow these questions

Large language models are trained on a corpus of text and are capable of emitting arbitrary amounts of text with similar statistical properties to their training corpus. These models usually have a temperature parameter which controls the randomness of the output, and this is usually tuned to be in a range where it is low enough that the output of the model is reasonably coherent, but also high enough that the output does not just reproduce extended portions of its training corpus.

This means that when asked for a quotation, there are good reasons to expect that the model has actually output a stream of text that only resembles a quotation, and when asked for a source for the quotation, there are good reasons to expect that the output only resembles a source.

Therefore, considering the current state of the art of these models, it is a waste of time for the Literature Stack Exchange community to attempt to locate the non-existent source of text that has been output by a language model.