I’m doing a claim verification exercise in a class tomorrow and I will use this. I’m glad you included SIFT but I think the addition of follow the funding makes it a lot stronger.
I’ve done something similar with a video on ‘alien’ Peruvian mummies and a video put out by gaia.com of all people. Students are too young to remember when they started as a yoga equipment company in the 90s lol.
As a longtime Biotech researcher, there is an additional layer to understanding content that comes from Biotech or Pharma companies: the closer something is to either a regulatory submission or to a market-moving clinical update, the less likely they are to lie because they can get into big legal trouble by lying.
However, they may word truthful statements about new clinical data in ways that make it tricky to tell whether it's "good news" from the company's perspective or "bad news." Sometimes the quickest way to tell is by looking at what happened to the company's stock price in the hours right after a press release.
Approved product labels are also required to be accurate. Marketing materials can have spin that is illegal in labels. But the labels are written in a medical-worker-friendly format, not a patient-friendly format.
Question: it's a truism that an LLM "knows" only the works it's been trained on. (I recently asked Claude when its training was last updated; it said August of 2025.) During the training, do the developers assign any kind of, well, call it "veracity weighting" to a given source -- based on the nature of the source (academic, mass market, whatever)? And for that matter, to the extent that the training focuses on linguistic patterns rather than specific facts -- truths -- does one sort of pattern (academic and peer-reviewed, journalistic presentation, whatever) get weighted over another?
This is a useful framework, Hana, and I especially appreciated the distinction between verifying a claim and exploring a subject.
The practical difficulty, to me, is the time and expertise involved. Even as a scientist, I cannot independently assess everything I read. Your section on source credibility seems especially important here: we often need someone to help interpret the evidence, but then verification changes target.
I would look for someone who distinguishes evidence from interpretation, acknowledges uncertainty, and corrects mistakes. But credentials and strong communication skills aren't enough on their own. What would you suggest as a manageable way for readers to assess whom to rely on?
I like to refresh my understanding of what "research" means every once in a while because living in the digital age it is so easy to get stuck in a feedback loop (confirmation bias). Excellent article miss Hannah, I thank you for being that reminder this time.
Great article Hannah. I have a friend from finance who is doing a later life PhD and I’ll share with him. Research skills have been neglected by universities here in the U.K. in favour of what I call citation sifting. Sometimes you need to read bad or dodgy research before you get to the credible end. However in my experience ( returning for a masters in later life after earlier Masters and PhD research), , masters students in economics and finance and sometimes even psychology are pushed towards cannon literature too early. There is also a worrying reliance by masters students themselves on popularisers who may communicate well but may not be the most rigorous. I’ll capture your insights for my team also.
The HRT section landed for me, that gap between what the WHI actually found and the four words everyone kept. But the part that nags me is you can go through every step here and still come out confirming what you already thought. The checklist doesn't catch that. I mean you don't catch it, that's the problem, you feel like you did the work. Enke had people do cognitive tasks for what amounted to a month's rent and they just, they tried harder and got the same answers wrong. Took 40% longer about it.
I'd put one more thing before follow the funding. Write down what would make you completely wrong. Not what would complicate things, what would demolish them.
Wason's people were careful by the way. That was the problem.
I’m doing a claim verification exercise in a class tomorrow and I will use this. I’m glad you included SIFT but I think the addition of follow the funding makes it a lot stronger.
I’ve done something similar with a video on ‘alien’ Peruvian mummies and a video put out by gaia.com of all people. Students are too young to remember when they started as a yoga equipment company in the 90s lol.
As a longtime Biotech researcher, there is an additional layer to understanding content that comes from Biotech or Pharma companies: the closer something is to either a regulatory submission or to a market-moving clinical update, the less likely they are to lie because they can get into big legal trouble by lying.
However, they may word truthful statements about new clinical data in ways that make it tricky to tell whether it's "good news" from the company's perspective or "bad news." Sometimes the quickest way to tell is by looking at what happened to the company's stock price in the hours right after a press release.
Approved product labels are also required to be accurate. Marketing materials can have spin that is illegal in labels. But the labels are written in a medical-worker-friendly format, not a patient-friendly format.
Your essays are fabulous, Hana! Thanks for such high-quality work! 👏
THANK YOU. Wonderful essay.
I've been wanting to write an article just like this months! Well done.
Outstanding as always!
Question: it's a truism that an LLM "knows" only the works it's been trained on. (I recently asked Claude when its training was last updated; it said August of 2025.) During the training, do the developers assign any kind of, well, call it "veracity weighting" to a given source -- based on the nature of the source (academic, mass market, whatever)? And for that matter, to the extent that the training focuses on linguistic patterns rather than specific facts -- truths -- does one sort of pattern (academic and peer-reviewed, journalistic presentation, whatever) get weighted over another?
This is a useful framework, Hana, and I especially appreciated the distinction between verifying a claim and exploring a subject.
The practical difficulty, to me, is the time and expertise involved. Even as a scientist, I cannot independently assess everything I read. Your section on source credibility seems especially important here: we often need someone to help interpret the evidence, but then verification changes target.
I would look for someone who distinguishes evidence from interpretation, acknowledges uncertainty, and corrects mistakes. But credentials and strong communication skills aren't enough on their own. What would you suggest as a manageable way for readers to assess whom to rely on?
"I've done my research" comeback:
https://u.pcloud.link/publink/show?code=XZ6CYkJZEqcUwYKJCOXO2Ny2Ct8XLQdTYwkX
https://youtu.be/nIw8nnVOQd4?is=tYMu56H2WyegcA5B
Maravillo ensayo, super útil y super valioso. Thanks!
Primary sources are the best.
I like to refresh my understanding of what "research" means every once in a while because living in the digital age it is so easy to get stuck in a feedback loop (confirmation bias). Excellent article miss Hannah, I thank you for being that reminder this time.
Great article Hannah. I have a friend from finance who is doing a later life PhD and I’ll share with him. Research skills have been neglected by universities here in the U.K. in favour of what I call citation sifting. Sometimes you need to read bad or dodgy research before you get to the credible end. However in my experience ( returning for a masters in later life after earlier Masters and PhD research), , masters students in economics and finance and sometimes even psychology are pushed towards cannon literature too early. There is also a worrying reliance by masters students themselves on popularisers who may communicate well but may not be the most rigorous. I’ll capture your insights for my team also.
Great essay! Thanks.
The HRT section landed for me, that gap between what the WHI actually found and the four words everyone kept. But the part that nags me is you can go through every step here and still come out confirming what you already thought. The checklist doesn't catch that. I mean you don't catch it, that's the problem, you feel like you did the work. Enke had people do cognitive tasks for what amounted to a month's rent and they just, they tried harder and got the same answers wrong. Took 40% longer about it.
I'd put one more thing before follow the funding. Write down what would make you completely wrong. Not what would complicate things, what would demolish them.
Wason's people were careful by the way. That was the problem.