truHug's guide to interpreting research for yourself
Hello all, Jonathan from truHugs here. With all the scientific research on weighted blankets out there, sometimes it may confusing as to what all these articles mean. How do we tell whether weighted blankets really work to reduce insomnia, stress, and anxiety? Summarizing scientific studies and applying them to a product isn’t just difficult for obvious reasons, like understanding and then explaining scientific jargon or methods to non-specialist. Often, the context gets completely lost when we translate findings into stories, tips, and tools for a more meaningful life, especially when we push it all through the nuance-squashing machine of the internet. Many never read past the headlines, which intrinsically aim to overgeneralize and provoke interest. Because resources that lack validity like blog posts and non-scientific articles can never be as comprehensive as the original scientific studies they are inferred from, they almost always omit some crucial caveats, such as limitations acknowledged by the researcher. To get those, you need access to studies themselves.
Adding to the problem is it’s very common for differing studies to contract each other. Different primary investigators of weighted blankets may have different preconceptions, biases, motivations, and incentives which modify how the study is conducted and what ultimate outcome is derived.
Journalists and members of the general public are mistakenly prone to treat each new study as though it represents the last input on the question addressed. In science, each study has its own limitations. It is for this very reason that it takes many studies examining the same phenomenon making the same conclusion to eventually establish a fact with the passage of time. In our ‘research’ section, we have already assessed and summarized much of the science behind weighted blankets. If you have the time, we encourage you to investigate each article yourself. Below are some of the questions you should ask when evaluating scientific articles examining the efficacy of weighted blankets.
- Where did the study come from? Did the study appear in a peer-reviewed journal?
Peer review is the rigorous process of having specialists in a similar field as a research review a scientific article before it’s published. These reviewers make sure what is written and what was done match, including conclusions drawn by the primary author. If the study was published in a non-peer reviewed journal, it’s credibility decreases dramatically, as what was presented could possibly be the highly subjective opinion of someone who lacks knowledge of how to conduct a study properly. If the article appeared in a non-scientific journal, like BusinessInsider, be careful of the claims made within the article. They could be making generalized claims without listing the limitations of the study for marketing purposes.
Who was studied? Where did the study take place?
In life, there are usually two elements in any given situation that will determine the outcome. First is your personal characteristics, aspects of yourself such as aesthetic appeal, age, gender, educational status, proneness to stress and special disabilities. A weighted blanket may work for someone of a particular disability, such as autism, but be ineffective in helping reduce a statistically significant difference in anxiety for a normal person. Second is the immediate context that surrounds you such as your immediate environment, how stressful your job is, how your friends may judge you for specific actions, and culture. As a brief example, a weighted blanket may work to reduce anxiety for someone whose lifestyle doesn’t entail a lot of pressure therapy or engage in required amounts of human contact, such as that of a computer programmer. However, an overwhelmed personal trainer may notice less of an effect. In order to determine whether weighted blankets would have the same outcomes for you, examine how similar the participants and the context within which they were studied are similar to yours.
How many subjects were in the study?
Needless to say, the more subjects that were studied whose context and personal characteristics match yours, the more likely the study outcome applies to you.If there were over 800 people your age, gender, and socioeconomic status with similar contexts as you (living in America dealing with similar stressful conditions, not broke in poverty struggling to survive), you can almost rest assured that the finding applies to you as well. If there were only 5 people used to derive the study outcome, you may benefit by questioning whether outcomes derived from studying such a minute number of people apply to the whole population.
Did the experimental design within a study control for confounding variables?
Let’s say a weighted blanket study had all females in the group that was assigned the intervention of using a weighted blanket whereas the control group only had males.The study concluded weighted blankets reduce anxiety and stress for everyone. A confounding variables are unaccounted for influencers of the outcome. The study may conclude weighted blankets caused the decrease in stress for a person while in reality, the only people who used a weighted blanket and decreased their stress was females. We cannot discount for the fact that the fact everyone who used the weighted blanket was female, and should not generalize the effect of reduced stress to males. Another example is if the weighted blankets were studied while relaxing spa meditative music was played in the background. Is it the music making people more relaxed or the blanket itself? Was the data collector an attractive member of the opposite gender? If so, when surveyed, male participants may report a decrease in anxiety simply due to implicit sexual attraction to the researcher. This leads into the question…
Was there a control group?
One of the first things to look for in methodology is whether the sample is randomized and involved a control group; this is especially important if a study is to suggest that a certain variable might actually cause a specific outcome, rather than just be correlated with it (see next point).
For example, were some in the sample randomly assigned a specific meditation practice while others weren’t? If the sample is large enough, randomized trials can produce solid conclusions. But, sometimes, a study will not have a control group because it’s ethically impossible. (Would most people kill one crazy person in order to save five lives, if their decision killed a real person, instead of just being a thought experiment? We’ll never know for sure!)
The conclusions may still provide some insight, but they need to be kept in perspective.
What were the methods used? Are the methods shown to be reliable, valid, and credible?
In clinical health studies, there are many tools and procedures used to measure variable outcomes. Tools and procedures used to measure outcomes must be shown to be reliable (the same outcome can be derived in multiple trials), valid (the tool must measure the intended outcome accurately), and credible (the tool or process has been shown by other studies to measure the outcome of interest). An example of this is a study that concludes there was a decrease in stress from weighted blanket use because participants had a lower heart rate after. While a lower heart rate may implicate a decrease in stress, it could also just mean the person went from a standing position to a lying / prone position.
In order to assess whether their methods for evaluating a “reduction in stress” are reliable, look for: (a) whether trials were repeated for different sample groups deriving the same outcome in the study, and (b) whether other studies derived similar outcomes with the same methodology.
In order to assess whether their methods for evaluating a “reduction in stress” outcome are valid / credible, look into the methodology section and look for: (a) how many other studies used their specific method of decreased stress, (b) how many studies have examined the direct relationship between reduced stress and decreased heart rate, (c) what factors in addition to reduction in heart rate may implicate decreased stress that were left unexamined by the study of interest, and (d) what might decreased heart rate represent other than a decrease in stress. Usually, these concerns are highlighted in the ‘methods’ section of a scientific article.
Of particular interest are qualitative (non-numerical studies) that seek to establish general perception of a product or topic. In order to determine if their qualitative methods were correct ask yourself: (a) how much did the researchers control their own biases when surveying for data(did the investigators used a validated survey form or their own created form, which may lead those surveyed to a particular response?), and (b) how many different ways did the researchers arrive at their proposed conclusion (did they know weighted blankets were more effective from participants own opinion and observations of the participants friends?).
Is the journalist, scientist, or business person overstating the outcomes of the experiment?
Language that suggests a fact is “proven” by one study or which promotes one solution for all people is most likely overstating the case. As mentioned prior, it takes many studies examining the same phenomenon to claim something as “theoretical fact.” This is because each study has to have their own design limitations, as you can see through points 2 through 6.
Currently, other than large scale government epidemiological studies, there are very few studies that examine the effects of an intervention on the entire global population. This makes sense, as such studies would be impractically expensive, time-intensive, and nearly impossible to implement. That is the type of study that would be required to state the exaggerated outcomes many businesses use to market their product.
Is there a conflict of interest suggested by the funding or the researcher’s affiliations?
A recent study found that you can drink lots of sugary beverages without fear as long as you exercise. The funder? Coca Cola, which eagerly promoted the results. This doesn’t mean the results are wrong, but you should seek opinions from parties who do not have a vested interest in the result, such as academic institutions.
Do the researchers acknowledge limitations and entertain alternative explanations?
With all the factors that could skew interpretation of a study’s outcome (listed points 2 – 7), does the researcher address how the study was approached and emphasize the limitations of the study of interest? Does he demonstrate awareness of which questions are answered and which aren’t by his methodology.
All of this sounds great, but it sounds like a time-intensive tedious process. If you believe a weighted blanket purchase does not warrant this much time investment because the costs do not outweigh the benefits, please check out our conclusions on weighted blankets. For those who do have three to four hours to invest in learning about weighted blankets, please visit our section on how to evaluate scientific research efficiently.