Improving psychological research

When psychologists repeated a hundred studies in 2015, their results differed in two-thirds of the studies. ‘Research into research is not a luxury but a necessity,’ says Professor of Methodology and Statistics of Psychological Research Mark de Rooij. ‘My aim is to improve psychological research, to think carefully about each step of the research to ensure that the ensuing research is better: from research question to publication.’

Computer simulation: a fictional Netherlands

De Rooij’s research group uses computer simulations to improve statistical research. They simulate the real world in a computer programme. Here they create fictional populations with specific characteristics. ‘I can create a fictional Netherlands, where I know the relationship between my predictor and my result. I therefore know what the truth is.’

What is true knowledge?

In these verifiable populations, De Rooij tests whether the statistical methods that psychologists use for research into, say, neuroticism or depression are correct. This can result in complex calculations. ‘Sometimes my computer takes one-and-a-half hours to complete 5,000 analyses where I take 100 samples from each of 50 populations. Other times such research entails leaving all the computers in the Faculty running from closing time on Friday evening to early on Monday morning.’
The results of such research tell De Rooij if the statistical technique really does find what it should find. This provides insight into whether the sampling and analysis methods used allow one to make general conclusions about the population as a whole. This helps nudge the results of psychological research into the realms of reliable knowledge.

Complexity of social sciences

Every study does have a margin of error. ‘What is special about psychology and the social sciences is that a lot of things can’t be directly observed. I can measure a person’s height, but I can’t see the extent to which that person is depressed. A psychologist uses a particular psychological instrument to measure this, a test or a questionnaire for instance, but these are never 100% accurate. There are always measurement errors, because everyone is different in every situation. That is what makes social sciences research so tricky and complex. No one person is exactly average, but in research we draw conclusions about characteristics of the supposed average person.’

100% reproducible

The computer simulation gives De Rooij access to a stable population. Here he can repeat the technique until there is almost no margin of error. ‘The research that I do is 100% reproducible. Someone in Bangladesh or New York can do the same thing and get exactly the same results. The knowledge that this yields is therefore true knowledge.’

De Rooij’s research group uses this knowledge to draw up guidelines that tell psychologists whether they can use the statistical technique that the group tested in their research and whether the analysis of their data will actually tell them what they want to find out. This improves psychological research and produces knowledge upon which we can continue to build.