���i Don/'t Know What To Believe...���
Project1 15/11/05 15:34 Page 1
sense about science
“I DON’T KNOW
WHAT TO
BELIEVE...”
Making sense of science stories
This leaflet is for people who follow debates about science and
medicine in the news. It explains how scientists present and judge re s e a rch and
how you can ask questions of the scientific information presented to you.
sense about science
SUMMARY
• Science has a system for assessing the quality of research before
it is published. This system is called peer review.
• Peer review means that other scientific exper ts in the field
check research papers for validity, significance and originality –
and for clarity.
• Editors of scientific journals draw on a large pool of suitable
exper ts to scrutinise papers before deciding whether to publish
them.
• Many of the research claims you read in newspapers and
magazines, find on the internet, or hear on television and the
radio are not published in a peer-reviewed journal.
• Some of this research may turn out to be good but much of it is
flawed or incomplete. Many repor ted findings, such as claims
about “wonder cures” and “new dangers”, never come to anything.
• Unpublished research is no help to anyone. Scientists can’t repeat
or use it and as a society we can’t base decisions about our public
safety – or our family’s health for example – on work that has a
high chance of being flawed.
• So, no matter how exciting or compelling new scientific or medical
research is, you must always ask…
Is it peer reviewed? If not, why not?
If it is peer reviewed, you can look for more information on what
other scientists say about it, the size and approach of the study and
whether it is par t of a body of evidence pointing towards the same
conclusions.
sense about science
HOW SHOULD YOU MAKE SENSE OF SCIENCE STORIES?
Ever y day we are bombarded with information about science from newspapers, radio
and television programmes and the internet. Making sense of it all can be ver y
difficult. What should be taken seriously? Which are ‘scares’? Sometimes scientists
are repor ted as saying conflicting things. How do we know what to believe?
There is a system used by scientists to decide which research results should be
published in a scientific journal. This system, called peer review, subjects scientific
research papers to independent scrutiny by other qualified scientific exper ts (peers)
before they are made public.
Peer review can help you make sense of science stories as it tells you that the
research has passed the scrutiny of other scientists and is considered valid,
significant and original.
Peer review means that statements made by scientists in scientific journals are
critically different from other kinds of statements or claims, such as those made by
politicians, newspaper columnists or campaign groups. Science is therefore more than
just another opinion.
A SHORT EXPLANATION OF PEER REVIEW
When a researcher, or team of researchers, finishes a stage of work, they usually write
a paper presenting their methods, findings and conclusions. They then send the paper
to a scientific journal to be considered for publication.
If the journal’s editor thinks it is suitable for their journal they send the paper
to other scientists who research and publish in the same field asking them to:
• Comment on its validity – are the research results credible;
are the design and methodology appropriate?
• Judge the significance - is it an impor tant finding?
• Determine its originality - are the results new?
Does the paper refer properly to work done by others?
• Give an opinion as to whether the paper should be published,
improved or rejected (usually to be submitted elsewhere).
This process is called peer review. The scientists (peers)
assessing the papers are called referees or reviewers.
Scientists never draw firm conclusions from just one paper or set of results.
They consider the contribution it makes in the context of other work and their
own experience. It usually takes more than one research paper for results to be
seen as good evidence or accepted as a public truth.
sense about science
THE SCIENCE PUBLISHING SCENE
For scientific knowledge to progress scientists need to share their research findings
with other scientists. The main way they do this is by publishing their research in
scientific journals – periodical publications intended to fur ther the development of
science by repor ting new research.
Journal editors receive many more papers than they can publish, so they use a two-
step selection process. First, they consider whether the paper is a ‘fit’ for their
journals. For example, some journals only publish research papers that are
groundbreaking; others only publish research in a specific area, such as microbiology.
If a journal editor decides that a paper is right for their journal, they send it for peer
review to check whether the research findings are valid, significant and original.
A NOTE ON JOURNAL
Did you know?
FUNDING AND
There are around 21,000 scholarly
AVAILABILITY
and scientific journals that use the
peer-review system. A high proportion
Most journals receive their income
of these are scientific, technical or
from subscriptions and some from
medical journals, publishing over
organisational subsidies, conference
1 million research papers each year.
organising and advertising.
Many are available on the internet
and, increasingly, make their
online content free after a certain
period, usually one year.
There are alternative journal
funding models, such as scientists
paying the costs of reviewing and
publishing their articles so that they
can be made freely available. Less
than 1% of papers are published
this way.
sense about science
Publishing in a journal is an integral par t
of being a scientist.
HOW CAN YOU TELL
It:
WHETHER REPORTED
• Connects like-minded individuals and
RESULTS HAVE BEEN
tells them about new research.
PEER REVIEWED?
A published paper is read by scientists
It can be difficult!
all around the world.
The full reference to peer-reviewed
• Is a permanent record of what has
papers is likely to look like this:
been discovered, when and by which
Fellers J H and Fellers G M (1976)
scientists – like a cour t register for
Tool use in a social insect and its
science.
implications for competitive
• Helps scientists to promote their work
interactions. Science, 192, 70-72.
and gain recognition from funders and
…or this:
other institutions.
Hedenfalk I, Duggan D, Chen Y, et al.
• Shows the quality of the scientist’s
Gene-expression profiles in
work: other exper ts have rated it as
hereditar y breast cancer.
valid, significant and original.
N Engl J Med, 2001; 344: 539-48.
By the way...
A few unscr
Peer review of research proposals
upulous
people use this style on websites
Peer review is also used to assess
and in ar ticles to cite work that is
scientists’ applications for research
not peer reviewed. But for
funds. Funding bodies, such as medical
tunately
this is rar
research charities, seek expert advice
e.
on a scientist’s proposal before
agreeing to pay for it. Peer review in
this instance is used to judge which
You are most likely to hear about
applications are the best science and
new research from the daily news
have the potential to help the
media, where there is not space or
organisation achieve its objectives.
interest in full references. Good
journalists usually indicate whether
research has been published and
mention the name of the journal.
sense about science
SO SCIENTISTS USE PEER
You can also look for longer repor ts
REVIEW, SO WHAT?
of the same research in other
newspapers, or popular science
When research findings have been peer
magazines, many of which are
reviewed and published in a scientific
online, to find out whether research
journal, this indicates that they are
is published and where.
sufficiently valid, significant and original
to merit the attention of other scientists.
This also helps for clarifying whether
the repor ted claims are a true
Peer review is an essential dividing line
reflection of the findings in the
for judging what is scientific and what is
research paper.
speculation and opinion. Most scientists
make a careful distinction between their
peer-reviewed findings and their more
Research papers pr
general opinions.
at scientific confer
esented
begun a pr
ences have often
Sounds good, but what happens
ocess of peer r
ar
eview but
e usually still unpublished and
next?
pr
Publication of a peer-reviewed paper is
eliminar y.
just the first step: findings, and theories
about them, must go on to be re-tested
and judged against other work in the
The more we ask, ‘is it peer
same area. Some papers’ conclusions will
reviewed?’ the more obliged
be disputed or further research will show
reporters will be to include this
that they need to be revised as more data
information.
are gathered.
There is no definitive list of peer-
reviewed journals but you can look
Just as a washing machine has a
up the names of selected peer-
quality kite-mark, so peer review is a
reviewed journals online at the
kind of quality mark for science. It
science news ser vice EurekAler t!
tells you that the research has been
(www.eurekalert.org/links.php?jrnl=A)
conducted and presented to a
Sources of fur ther help with
standard that other scientists accept.
ascer taining the status of
research are listed at the end
of this leaflet.
sense about science
CHALLENGES FOR PEER REVIEW
Why can’t there just be a checklist of
scientific validity?
Assessing scientific papers cannot be done in the
same way as giving a car an M.O.T. or marking a
maths test. New research usually has its own
unique features, which are difficult to predict with
a check list and which require exper t judgement about
their validity, significance and originality.
Does peer review detect fraud and misconduct?
Peer review is not a fraud detection system. Referees are likely to detect
some wrongdoing, such as copying someone else’s research or misrepresenting data,
because they care about their subject. They know what research has been conducted
already and the kinds of results that are likely. However, if someone deliberately sets
out to falsify data, there is sometimes no way of knowing this until the paper is
published and others in the scientific community scrutinise and tr y to repeat the work.
Is ‘maverick’ science rejected through peer review?
Sometimes people worr y that new ideas won’t be understood by other scientists
(although this is also an excuse given when researchers don’t want to submit to the
scrutiny of their peers). It is true that referees can be cautious about unusual findings;
and impor tant insights can initially be overlooked. But if someone has been
exceptionally clever, other scientists are most likely to recognise it and to distinguish it
from flawed or inflated claims. Journal editors like novel ideas and scientific publishing
has brought thousands of impor tant discoveries to light.
Does the peer-review process slow down advances in scientific and
medical knowledge?
In our world of instant communication and 24-hour news, a deliberative process like
peer review can seem frustratingly slow. Electronic communication has improved it, but
good assessment of research does take time. Sometimes people justify the promotion
of unpublished findings by saying they are ‘too impor tant to wait’. But, although some
papers take months to review and improve, if there is a major breakthrough the
process can be completed in weeks. Fur thermore, if the findings are ver y impor tant –
e.g. they concern public health – then it is all the more necessar y to check them
through peer review.
sense about science
SOURCES FOR FURTHER INFORMATION
Sense About Science: To find out more about peer review you can visit the Sense About
Science website where there is a section dedicated to it. The section includes free
downloads of Sense About Science’s longer repor t: Peer Review and the Acceptance of New
Scientific Ideas (2004), electronic versions of the leaflet and additional educational
resources. To request fur ther copies of the leaflet please email:
publications@senseaboutscience.org or call: +44 (0) 20 7478 4380
www.senseaboutscience.org/peerreview
Association of Medical Research Charities: The AMRC has a page, for medical research
charities, on the peer review of research grant applications:
www.amrc.org.uk/temp/Aboutsppeerspreview.doc
Committee on Publication Ethics: COPE provides a sounding board for journal editors
struggling with how to deal with breaches in research and publication ethics:
www.publicationethics.org.uk
The National Electronic Library for Health: The NELH has a ‘Hitting the Headlines’ archive,
which looks at medical news stories and provides the research evidence on which they are
based: www.nelh.nhs.uk/hth/archive.asp
The Science Media Centre: The SMC has published a leaflet, Peer Review in a Nutshell, a
guide for scientists preparing for a news inter view.
www.sciencemediacentre.org/peer_review.htm
ACKNOWLEDGEMENTS
This leaflet has been produced and distributed with sponsorship and help from:
Sense About Science is grateful for the input of the sponsors, the many organisations (in
par ticular Cancer Research UK, Asthma UK, Migraine Trust and Action Medical Research),
parliamentarians, government officials, educational organisations, teachers, school
students, doctors, pharmacists, science bodies and the many others, who kindly contributed
their time and ideas. Responsibility for the content rests fully with Sense About Science.
The Lancet cover reprinted from The Lancet, Vol. 366, No. 9487, 27 August 2005, with permission
from Elsevier.
Science cover reprinted with permission from AAAS.
© Sense About Science 2005
Sense About Science, 60 Cambridge Street, London SW1V 4QQ
Registered charity number 1101114
