There’s a new article out that uses faulty methods to study the linguistic complexity of politicians’ speech. It makes many of the same mistakes that I criticized Schoonvelde et al. (2019) for – and even references that article. But it somehow comes to the right conclusion… for the wrong reasons. I know, it’s strange. Let’s check it out. Continue reading “More political scientists doing bad linguistics”
Tag: Flesch-Kincaid
Update on that F-K paper
Three months ago I posted about a paper in PLoS ONE called “Liberals lecture, conservatives communicate: Analyzing complexity and ideology in 381,609 political speeches”. I noted that there are serious problems with that study. For the tl;dr:
After I posted on here, I also commented on the article with my concerns. The PLoS ONE journal allows commenting on their articles, but I’ll admit that my first comment was neither appropriate nor helpful. It was more of a troll than anything. The editors removed my comment, and to their credit, they emailed me with an explanation why. They also told me what a comment should look like. So I posted a grown-up comment on the article. This started an exchange between me and the authors of the article. Here’s the skinny:
1. The authors confuse written language with spoken language
2. The study uses an ineffectual test for written language on spoken language
3. The paper does not take into account how transcriptions and punctuation affect the data
4. The authors cite almost no linguistic sources in a study about language
5. They use a test developed for English on other languages
The authors tried to respond to my points about why their methodology is wrong, but there are some things that they just couldn’t argue their way out of (such as points 1, 2, 3 and 5 above).
Behind the scenes, I was talking with the editors of the journal. They told me that they were taking my criticisms seriously and looking into the issue themselves. In my comments on the paper, I provided multiple sources to back up my claims. The authors did not do in their replies to me, but that’s because they can’t – there aren’t studies to back up their claims. However, my last email with the editors of the journal was over a month ago. I understand that these things can take time (and the editors told me this much) but a few of the criticisms that I raised are pretty cut and dry. The authors also stopped replying to my comments, the last one of which was posted on April 9, 2019 (can’t say I blame them though).
So I’m not very positive that anything is going to change. But I’ll let you know if it does.
Stop using the Flesch-Kincaid test
Before Language Log beats me to it, I want to hip you to another Bad Linguistics study out there. This one is called “Liberals lecture, conservatives communicate: Analyzing complexity and ideology in 381,609 political speeches” and it’s written by Martijn Schoonvelde, Anna Brosius, Gils Schumacher and Bert Bakker. It was published in PLoS One (doi:10.1371/journal.pone.0208450).
The study analyzes almost 400,000 political speeches from different countries using a method called the Flesch-Kincaid Grade Score. The authors want to find out how complex the language in the speeches is and whether conservative or liberal politicians use more complex language. But hold up: what’s the Flesch-Kincaid score, you ask. Well, it’s a measure of how many syllables and words are in each sentence. The test gives a number that in theory can be correlated to how many years of education someone would need in order to understand the text. This is called the “readability” of the text.
So what’s the problem? Well, rather than spend too much time on it, I’ll listicle-ize the problems with this paper.