Appeal to authority: I'm a professor at Harvard, I'm a world expert on video games and kids, and I think you shouldn't let your kids play video games (because my gut tells me its bad, but I won't admit that on camera).
Citing your sources: I'm Joe the plumber, and I don't let my kids play video games because I read three papers in PNAS (here they are) that showed video games worsen ADHD.
Now, my big beef is that "appeal to authority" often pollutes published sources ("citations").
Reviewer: Hey editor, paper looks ok (The results of this paper don't match with the conclusions, and there are some gaps in the controls, but who am I to argue with the big professor from Harvard who's been publishing papers on this topic for 20 years and make her my enemy.)
Editor: Okey Dokey. (This paper will get lots of clicks and citations and make my journal look cool.)
And even worse
Reviewer: Hey editor, this paper is bad (The results go against what I've been publishing for 20 years, and I can't actually find a problem with it, but I'm the big shot here, so I say no.)
Editor: Okey Dokey, (I don't want to piss off the big professor from Harvard and get on the bad side of the academic gossip train.)
I used to be a researcher (in surface engineering and coatings technology, nothing very interesting though we had some amazing machines, but still...) and we were sent of courses to teach us about peer review and citation that basically told us : 'Yes, this is a thing, the second example more than the first - be very very wary'.
OK, the big point here is: The sources you cite, or journal you publish in, aren't the test for your model-of-how-things-work and your argument. The test is other people repeating your experiment in the real world, and doing variations on it, and seeing the same results that you get, and not being able to propose an obviously less complex model that makes seriously better predictions of further results. That's the function journals, and so citations in part, fulfil - communicating what you did so others can test it. Even if what you did stands scrutiny, don't be surprised if others push for revisions, for adding new limits to the use of your model (you should already be being up-front about where it's limits are, and it definitely does have them whatever it is) and other changes that are less severe than totally knocking your work over.
If you just want the really important point about what makes scientific research useful to the world if done right, you can stop reading here. The rest is pretty boring, though important to me as an ex-researcher. That said....
Citing your sources isn't totally valueless. It's better than nothing. Citing your sources is, in theory, a good thing because it allows other people to check where you've got your ideas, approach, and contributing knowledge from.
BUT: It doesn't / shouldn't, by itself add much weight to your argument - it's just, a sign of good intention you might say. To the properly skeptical reader, again in theory, it's one of several means they should use to decide for themselves how much weight to give your argument - by checking at least the sources cited in support of any key, or possibly controversial, points. It's not that citing your sources should remove
any level of skepticism from your audience, it's that not citing sources on an important or controversial
should immediately add two layers of skepticism.
A few specific points:
The reviewers for your paper
should be kept anonymous from you.
In fact, to avoid people guessing who their reviewers are in a small field, the reviewing should be selected at random, and may not even be by someone in your field or a closely related one (obviously, this is its own problem). So you shouldn't risk making an enemy with an honest review.
Journals are anxious to preserve their reputation (because they often charge a bomb for full access), and there are organisations that act as watchdogs, so while duff reviewing and turning a blind eye (for various reasons) certainly takes place, it should stay at a low level.
Most research work is pretty dull 'attacking the boundaries of knowledge by gradual, mind numbing, attrition' stuff - it's just not interesting, or important, all by itself. Most editors wont see the title as an attention grabber, most academics wont see it as a threat even if it disagrees with them.
Last: Journalism is not good science. Good PR
very definitely isn't. What sells copy, what sounds pithy, is easily absorbed by every reader, and
especially what sounds gains popularity and is
cool (spits), is not a sign of truth. hence why I'm running on and on here. The biggest sh*tter is that 'what attracts funding' might not be either, unless the funding body is run by scientists who are still more scientist than manager or politician, despite being in charge of a funding body. '
Alien life discovered' as a front page Guardian headline does not mean we can tick that question off as answered scientifically. When so much alien life has been 'discovered', over many years, that it barely merits a mention on page 10 - then we can tentatively tick it off.
All the above is, absolutely, full of gaps and flaws. People gossip, turn favours, they get tribalistic about things, they pull dirty tricks and walk right up to important red lines and stick their toes over - sometimes because their career and livelihood and the future of their lab on the line, sometimes because their ego is enormous. Someone with the right contacts and influence who really wants to pull a fast one absolutely can, and stand at least a fair chance of getting away with it for some time.
I honestly don't have the skill or stomach to navigate through that stuff to success, which is why I left research. But, at it's heart, the issue is that there is no system anyone could construct that would be totally immune to that stuff - except for the old fail-safe that, eventually, you have your claims tested against how the world itself actually works, at which point they either hold up or they don't - and anyone who carries out your experiment will be able to see if they hold, or not, themselves.
That last 'test against the world' thing... even that's good in theory, and in practice, for only a majority of stuff. But like all great theories it also has its own well defined limits: What if your experiment uses the most powerful particle accelerator, or a detector that's one-or-two-of-a kind, or makes use of an opportunity that only occurs every 250 years?
So, in the end, at least part of it comes down to having faith that the vast majority of researchers, on any given day, really do care about finding the truth (with a caveat that they probably care more about keeping their kids fed), and are awake to there not just being the (IMHO fairly small) possibility of outright deception, but the much more common thing of honest mistakes, both of theory and practice and personal philosoph., And that they are willing to look for, and speak out against, at least the obvious, occurrences of them.
Sorry for going on and on but, once, many years ago now, this was my thing - and while I left, it remains close to my heart. Whiuch you can tell, because I have just unashamedly written a pretty dull essay on the philosophy of science...
