Reply to “Correspondence on ‘cover letters written by ChatGPT-4 or humans’“

Thank you for commenting [1] on our paper “A comparison of cover letters written by ChatGPT-4 or humans” [2]. While we appreciate the comment, it seems that some aspects of our research might have been overlooked.

Initially, it is important to mention that the results of the subjective evaluation of readability did not show that GPT-4 performed better than humans. Rather, the evaluation showed that GTP-4 letters were non-inferior to human-written cover letters, meaning that GTP-4 performed as well as humans, which was what the study aimed to explore. We agree that authors of cover letters bear the final responsibility for any draft written by AI, which was one of the reasons why we adopted a non-inferior design.

The sample size was, as always, a potential limitation, but sample size was determined by way of the sample size calculation, which was made from a thin base of assumptions since the study was one of the first of its kind, and we had no other research that had made similar comparisons that we might lean towards. We also acknowledge that increasing the number of assessors per cover letter could, in theory, contribute to more narrow effect estimates of the results. However, as in any study, we underlined by practical constraints and had to find independent researchers with medical and PhD degrees who would voluntarily spend their free time assessing these cover letters and completing questionnaires. Furthermore, it is important to clarify that we did not include only three assessors for the entire study. Rather, we engaged eight assessors and ensured that each cover letter was assessed by three different individuals.

We agree that other parameters, such as content and formatting, would have been relevant. However, our questionnaire was designed to evaluate impression, readability, criteria satisfaction and details that indirectly addressed these aspects. For example, criteria satisfaction included adherence to content standards, and readability and impression are influenced by the formatting and presentation of the cover letters. We thought that the included parameters were highly relevant, but others may choose to include or exclude different parameters.

Lastly, it is worth noting that future studies comparing other AI systems would be of great interest. However, that would have been a different study objective, and one should consider whether it would be relevant when ChatGPT-4 writes cover letters just as well as human authors who will always be responsible for the final version anyway. While we acknowledge the broader context of AI tools in professional writing, our study provides meaningful insights within its focused scope.