ChatGPT’s advice is perceived as better than that of professional advice columnists - Howe, Piers Douglas Lionel; Fay, Nicolas; Saletta, Morgan; Hovy, Eduard (Frontiers in Psychology, 2023)
ChatGPT is a high-performance large language model that has the potential to significantly improve human-computer interactions. It can provide advice on a range of topics, but it is unclear how good this advice is relative to that provided by competent humans, especially in situations where empathy is required. Here we report the first investigation of whether ChatGPT's responses are perceived as better than those of humans' in a task where humans were attempting to be empathetic. Fifty social dilemma questions were randomly selected from ten well-known advice columns. In a pre-registered survey, participants (N = 404) were each shown one question, along with the corresponding response by an advice columnist and by ChatGPT. ChatGPT's advice was perceived as more balanced, complete, empathetic, helpful, and better than the advice provided by professional advice columnists (all p-values < 0.001). Although participants could not determine which response was written by ChatGPT (54%, p = 0.29), most participants preferred that their own social dilemma questions be answered by a human than by a computer (77%, p < 0.001). ChatGPT’s responses were longer than those produced by the advice columnists (mean 280.9 words vs 142.2 words, p < 0.001). In a second pre-registered survey, each ChatGPT answer was constrained to be approximately the same length as that of the advice columnist (mean 143.2 words vs 142.2 words, p = 0.95). This survey (N = 401) replicated the above findings, showing that the benefit of ChatGPT was not solely due to it writing longer answers.