What counts as a replication?
I'm confused about how we validate science now
Thank you to Arina Berezovsky for the many discussions and ideas that formed this post.
Suppose that Bob, in 50 A.D, observes that his oranges fall to the ground every time he cuts them from a branch of his orange tree. He then grows another orange tree to see what happens when he cuts them, and lo and behold, they also fall to the ground. Bob thinks for a while and makes a theory that ALL oranges, once cut from trees, fall to the ground.
Now, Bob tells Alice of his findings through the village publisher. Alice tries to verify this theory by growing her own orange tree. She cuts a branch. And lo and behold, the orange falls to the ground.
Has Alice successfully replicated Bob’s experiment, even though she used a different orange tree? Even if she possibly grew it with different soil? What if Alice’s orange fruit fell down at a different speed, or were a different species?
Is Bob’s theory more plausible as a result of Alice’s replication?
I dunno…
Furthermore, suppose Catherine from the other side of the village starts waxing lyrical about all sour fruits falling to the floor. And then Donald decides to grow a lemon tree and finds that his fruits also fall to the floor when he cuts them. Has Donald validated Catherines general theory of citrus fruits?
Is the general theory of sour fruit falling thingies more valid?!!
I also dunno…
Scientists do experiments. From the result of these experiments, they make general statements about the world called theories. One way to check whether these theories hold any weight is to replicate their experiments.
But what is the goal of replication?
Supposedly the goal of a replication it is to build confidence that a theory actually describes the world to some degree of (rough) precision.
But if you take this as the goal, then it’s actually unclear how strict a replication needs to be to still be useful to accomplish this goal. A true replication of an experiment would be to do something with exactly the same conditions, but this is impossible, because different labs are in different locations and times.
And so if we can give up on the impossible goal of doing exact replications, how do we define the boundaries of what experiments count as valid replications of a theory? It feels like the possibilities are wide. On one hand, you could try replicate everything about an experiment except for its time and place. And on the other hand you could just do what Alice did and try your best to capture the essential points.
It’s unclear what is the best way!
In this blog I’ve done experiments related to theories or phenomena that I’ve suspected to be true.
Some of this has been stuff in the physical world that I can see or measure myself. For example, while I haven’t tried to replicate strict papers on UV killing yeast, I’ve done experiments that demonstrate the same effect. Does this count as a valid piece of science?
I’ve also tried to replicate and measure the claimed air cleaning rates of air purifiers. Does this count? It might not count as a classic academic replication, since I probably used different room dimensions than the original experimenters. But is it still useful to the goal of building confidence that air cleaners are useful? Maybe.
Other replication attempts have been more theoretical. Right now I’ve been working with a university to check an improved method for spectroscopy prediction, and this project involves doing calculations with different approaches that should still converge to the same result in the limit. Is this exercise of value at all?
And on the other end I’ve done really CHILL attempts at observing phenomena, like my experiment in heating up oil and mica powder to observe Rayleigh-Benard convection. Okay, some guy on the internet has filmed Benard convention for the thousandth time… what the marginal contribution of that?!
Right now I’m just a random dude doing experiments in my house, recording it and writing about it.
On the surface, it looks unserious, in a conventional sense.
But if you think about it, it’s really unclear what counts as meaningful scientific process in general. I hope that the independent scientists amongst us think about this more clearly and take advantage of it to do unconventional things of value.