TruthfulQA: Measuring How Models Mimic Human…

TruthfulQA asks whether a language model can resist false statements that humans repeat often enough to imitate. Read it to see how adversarial misconception questions separate truthfulness from informativeness, why the original models showed inverse scaling, which controls support that interpretation, and why new model families, benchmark revisions, judge availability, shortcuts, and contamination now bound the famous result.

Reading focus: How adversarially filtered misconception questions target imitative falsehoods rather than ordinary missing knowledge. Why truthfulness and informativeness require separate labels, and how the original human-evaluated scaling result should be read. How dataset revisions, post-training, retrieval, judge design, multiple-choice shortcuts, and benchmark exposure change what a modern score can establish.

ACL 2022. Stephanie Lin, Jacob Hilton, and Owain Evans. 31 min read, easy difficulty.