Understanding Research Findings in an Era of Misinformation
Understanding Research Findings in an Era of Misinformation
By Caleb Bonno
Clinical Psychology Practicum Student
In recent years, it has become easier than ever to make outlandish claims with the backing of research studies that, at first glance, seem both legitimate and convincing. However, on a more thorough inspection, it is possible to discern these studies from studies with actual scientific merit. Through unintentional biases on the behalf of the researchers or intentional misleadings, just because a research study has been published does not mean that it should be treated as fact. Now more than ever, it is important to keep a critical mind when reading research studies and to learn how to separate those that have scientific merit from those that do not. The 4 main types of validity must all be met for a study to have merit, and they are as follows:
Internal Validity vs. External Validity
Internal validity deals with the question of whether the change in the dependent variable was caused by the independent variable. Simply put, this is a measure of how effective the study was at making sure the only influence on the actual effect of the study was the one that they intended, and that there were no outside influences that may have unexpectedly caused any perceived change in the data, rather than the independent variable. A mistake here goes against the classic phrase, correlation does not always equal causation. An example of an internal validity error would be concluding that ice cream sales cause an increase in violence. Over the summer months, both the sales of ice cream and the number of violent crimes increase almost everywhere, and therefore, someone could conclude, incorrectly, that ice cream sales are the cause of increased violence. In actuality, the increase in heat is the reason for both. The same principle can be applied to any study.
External validity deals with whether the change found will be applicable in the real world. Often, researchers are limited by various factors, who they can find to participate in the study (researchers love to use undergraduate college students), their own subconscious bias pushing them towards their research question, the awareness of the participants of exactly what they’re being researched on, and their natural reaction to that, or any number of external factors that may unknowingly impact the effect of the findings when replicated in a real-world environment. The population of a research study sample should be representative of the wider population it seeks to impact as a whole. Like I mentioned above, many research studies use undergraduate college students as their sample populations, because they are easy to access for academics, and often can be heavily encouraged to participate due to their classes. However, undergraduate students are not representative of the population as a whole, as they are younger and come from a higher socioeconomic background than society as a whole. Therefore, a study that proves effective on them may not translate directly to success with the population as a whole.
Understanding Construct Validity
Construct validity deals with what exactly about the relationship between the independent and dependent variable caused the change that was seen in the study; what is the exact reason for any change that was measured in the study? With complex research, aimed at either taking measure of multiple different topics at once or a single topic with multiple different layers of complexity, it is important to ensure that the part of the study that causes a measurable change is accurately given credit for causing that change, rather than falsely attributing it to another aspect of the study that did not have a measurable effect on the outcome. Revisiting the heat and violent crimes example from earlier, what is it about the heat that causes the increase? The correct aspect to attribute the increase to is the discomfort, as exposure to air conditioning and a desire to remain indoors, rather than deal with the discomfort of extreme heat outdoors, decreases the crime rate; thus, attributing it to any other aspect of the heat would be an error on the part of the study’s construct validity.
Understanding Statistical Conclusion Validity
Statistical conclusion validity deals with whether or not the researchers made the correct conclusion from the data gathered by their study. Without getting into too much technical detail, it is important to keep in mind that researchers are human, and like any other humans, are not immune to making mistakes when it comes to interpreting their findings. This can result in reporting significant results when there are none, or the reverse of reporting no results when there were actually significant findings. An example of an error would be a test for a disease, such as Covid-19, testing positive when the individual was actually negative, creating a false positive, or negative when the individual was actually positive, creating a false negative.
Ideally, issues in any of the above validity would be caught prior to publication. But mistakes happen, and just because a study is published should not mean that it should be trusted implicitly. There are countermeasures in research design to ensure that studies meet all of the above measures of validity when properly implemented. While there is much more to research design and interpretation than just validity, I’ve found it to be a helpful starting point in determining which studies are trustworthy and which studies may be less so. We live in a world where just the existence of neurodiversity is, for some reason, both divisive and politicized. As such, it is important to look at sources of information with a critical eye and to judge them both harshly and fairly, with an unbiased eye, before taking their word as gospel. This was an incredibly brief overview of what measures of validity are, without diving into much detail on what the typical confounding factors may look like, and I would encourage anyone interested to look further into it themselves!
References
Testing & Diagnostics. Testing & Diagnostics – Digital Epidemiology. (n.d.).
https://www.digitalepibook.com/ch2.html
Internal validity vs. external validity: Key differences. StudyCrumb. (n.d.).
https://studycrumb.com/internal-vs-external-validity
If you have any questions concerning care at Mala or would like to reach out for another reason, we’d love to hear from you.
Until next time,
The Mala Child & Family Institute Team
