When The Scientific Method Fails
A physicist's reflections on living life well through the skill of judgement, not hard-core empiricism .
Frameworks for living better
I’ve been reflecting a lot on how to live life better. Whilst there are many ways to do this effectively, I now have some conviction that the best method is by making better decisions. Decisions seem to be the main channel to living better, since fairly motivated people generally have the same amount of working hours in the day. And so the only real differentiator you have have versus peers (with a similar background / environment that you had), is the choices you make. I enjoy figuring out ways to have more money, do better research, live a more comfortable life, and have better health and relationships. But how does one make better decisions? I think the answer to that is by improving your decision making framework.
And if the title isn’t already a hint enough, this post is going to be about some errors in my own decision making framework in the past.
The Scientific Method
As someone who does science research, and as someone who has been trained in math and physics, I’ve had some prior ideas about what makes a good framework for making decisions. Initially thought that the scientific method was a universally good way to guide oneself to making better decisions. The scientific method, to first order, is a recipe book that was originally used to arrive at truths about nature. To first order, the steps go like this:
make or guess a hypothesis about what is true, to come up with an updated ‘theory’ about the world. Example - diseases are caused by things called ‘germs’, that are really really small entities we can’t see but can be killed.
make sure that what is new in the model fits into your current existing model of what else is true in the world. Example - Since germs are really small, we can’t expect to see them with the naked eye, and so there is nothing obviously wrong with this germ theory.
figure out the novel thing that your theory predicts. Example - if we sanitise the rooms, we can kill the germs that cause disease!
if you’ve done 1., 2. and 3. correctly, congratulations! Now you have to verify that your new theory could be correct by testing the novel thing your theory predicts. Example - lets see what happens when we do surgery in dirty rooms versus clean rooms.
If your experiment turns out to be false, go back to step 1.
And I think sometimes, thinking about life problems with the scientific method is useful. Like in exercise, for example. For example, when was working on trying to work on my skills in calisthenics, I tried to use the scientific method quite liberally. I did fairly rigorous work in trying to isolate the problems that I had with my current technique, came to some ground truth and then experimented. It’s also a really useful technique obviously if you have a career in
engineering - stress testing designs
finance - backtesting investment hypotheses
or tech - unit testing software.
Where the Scientific Method Doesn’t Really Help
But unfortunately I think that the scope of applicability of the scientific method is just really small, and doesn’t apply to most of the things that are meaningful to me - even when, funnily enough, doing science!
Whilst the scientific method tells you how to test if something is true - it doesn’t tell you much about what you should test. For example, suppose I want to make an awesome research breakthrough. The scientific method doesn’t tell me at all about how to find and choose interesting and useful hypotheses. It only really tells me to test them. To find interesting and useful hypothesises, I’d have to defer to other methods decision making and thinking. For example,
finding a supervisor to guide me (people skills, persuasion and introversion)
building probabilistic priors about the likelihood of me succeeding at proving different hypothesis (decision theory, bayesian reasoning)
judgement about whether a problem is important (talking with people, estimation)
asking questions (idea generation)
Moreover the scientific method says nothing at all about the ethics or morals of decision making in any normative sense. For that kind of problem we’d need to defer to law, history and analytic philosophy - all which have styles of thinking that have nothing to do with experiments. And for good reason - it’s pretty much impossible to make serious scientific experiments to decide what is or is not societally good, unless you want to seriously breach some traditionally ethical guidelines.
This is where I really see some clear value in perhaps imprecise and rambling type thoughts, because those often fuel ideas about what to then test with the scientific method. For example, if I’m messing around with some math, I might come across a mistake that I find interesting, then follow that lead to actually testing something rigourous.
Another area where the scientific method fails is that most problems in the world aren’t ‘stationary’ - which means that the structure we are trying to probe changes with time. If something changes with time, it’s really hard to make meaningful experiments because by definition, an experiment requires that everything but one thing is kept constant. For example, if I’m trying to probe the structure of an atom, I am betting on the fact that the natural laws that govern the dynamics of this atom stay the same. Whereas, if I’m trying to make a definitive statement about human society with the scientific
It’s important to note here that I’m not claiming that the scientific method is wrong or not useful. All I’m saying just that one should beware about using it too liberally in trying to think about problems in the world. My point here is that the scientific method is hailed as a fix it all way of thinking in school, but it’s not. I think part of the reason think this is because I’ve always looked up great physicists, but then perhaps extrapolated too far. So this has been quite a painful conclusion for me to come to.
In excruciating detail, here are some other where the scientific method fails as a default way to make decisions.
the cost of doing experiments is high. I think this is a real consideration. Doing a proper experiment takes time and patience and cost. It is sometimes simply not worth trying to do an experiment every time, rather rely on educated guesses.
Randomness. Think about playing a poker game - it’s really hard to improve as a player because it’s hard tell if your poker playing process actually improves - because poker games involve a huge degree of randomness.
It doesn’t quantify uncertainty. Noise and randomness in decision making is inevitable, and sometimes the scientific method just proves indefensible against this.
It doesn’t take into account your preferences and utility. The scientific method doesn’t say anything about whether you’ll actually enjoy the results of whatever is being tested. That is a mysterious, and purely emotional call.
So what do we do instead?
So, I’ve just spent quite a bit of time rambling about the drawbacks of the scientific method. What are some alternatives for decision making? Well, I aim to write a whole new post on this but here are some ideas that I’ve been playing around with
decision theory - this is the economic approach where given a space of decisions, you try to map out the probability of the success of each one along with its reward, and then figure out the decision with the highest expected value
analytical reasoning - the approach where you use pure ol’ fashion logic to just try and reason out mechanically how decisions lead to outcomes. (If I do this then this, then this, then this will happen, and so forth.)
pros and cons - sometimes, the simplest solution is the best! Making a pros and cons list and then literally counting the number of pros versus the number of cons
seeking mentorship - sometimes, you just don’t know what you don’t know. And the best course of action is just to find a mentor that you want to emulate, and take advice from them.
not thinking at all - this can be fun! Go to a rave and enjoy mindless dancing!
Somehow, I think the thing that really underlies all of these different frameworks is figuring out a way to get the external world to match your internal preferences. And so I’d like to give a special mention to ‘being introspective’ as a really great way to think about making decisions.
Overall, there really isn’t a clear answer on how to best make decisions, and so I’d defer to the reader to decide what the best mix of thinking styles works for them!
> For that kind of problem we’d need to defer to law, history and analytic philosophy - all which have styles of thinking that have nothing to do with experiments.
What about thought experiments?
Dear Physics Enjoyer
Thank you. Allow me please to make the following comment.
Here, I present a short description of the statistical analysis of Aspect's experiment.
The x is angle(a,b) in [0,2π).
The, a, is Alice's instrument parameter vector. The, b, is Bob's instrument parameter vector.
The angle is measured in the plane orthogonal to the A-S-B axis. This “orthogonal to the A-S-B in the plane variation of x" is sufficient variation for understanding the statistics of the experiment. It is also physically valid.
Note furthermore that the experiment is embedded in classical probability theory. E.g. the law of large numbers to estimate the probability space behind the Bell correlation formula.
Aspect then requires:
1. cos(x)= P(x,=) - P(x,≠)
1a. P(x,=)=N(x,=)/N
1b. P(x,≠)=N(x,≠)/N
1c. N(x,=)=N(x,+,+)+N(x,-,-)
1d. N(x,≠)=N(x,+,-)+N(x,-,+)
1e. N=N(x,=)+N(x,≠)
2. cos(x)=1-2sin²(x/2), x in [0,2π)
3. P(x,=)+P(x,≠)=1
4. P(x,≠)=sin²(x/2), x in [0,2π)
The left-hand in 4. is a data probability (estimate) determined by nature. The right-hand isn't a probability function. It isn't monotone non-descending for x in [0,2π).
5. For x in [0,2π), the alleged associated probability density is f(x)=(1/2)sin(x).
5a. This alleged probability density is derived from 4. via f(x)=dF(x)/dx. But observe, f(x)=(1/2)sin(x), x in [0,2π), isn't a probability density. It isn't positive definite for x in [0,2π).
5b. This f(x) results in negative probabilities and other violations of Kolmogorov axioms.
5c. There are absolutely no classical probability model assumptions necessary for the presented statistical analysis in order to describe the entanglement correlation.
6. Obviously, x is a part of the random variable X describing the event “x,≠” in the measurement.
7. The x is entirely free to have any value in [0,2π), in order to maintain Einstein locality.
If an experiment violates Kolmogorov axioms, it's invalid. In fact, potential Einstein positive data is excluded from observation by requiring, e.g., a not positive definite probability density for that data. That's clearly a nonsensical requirement.
Aspect's experiment is statistically flawed.
Hence, also within science, the scientific method can fail.