Stop using the Flesch-Kincaid test

Before Language Log beats me to it, I want to hip you to another Bad Linguistics study out there. This one is called “Liberals lecture, conservatives communicate: Analyzing complexity and ideology in 381,609 political speeches” and it’s written by Martijn Schoonvelde, Anna Brosius, Gils Schumacher and Bert Bakker. It was published in PLoS One (doi:10.1371/journal.pone.0208450).

The study analyzes almost 400,000 political speeches from different countries using a method called the Flesch-Kincaid Grade Score. The authors want to find out how complex the language in the speeches is and whether conservative or liberal politicians use more complex language. But hold up: what’s the Flesch-Kincaid score, you ask. Well, it’s a measure of how many syllables and words are in each sentence. The test gives a number that in theory can be correlated to how many years of education someone would need in order to understand the text. This is called the “readability” of the text.

So what’s the problem? Well, rather than spend too much time on it, I’ll listicle-ize the problems with this paper.

Continue reading “Stop using the Flesch-Kincaid test”