What I Learned from 500 Independent Scientists
Independent science is about to have its day in the Sun again.
Why aren't more working people doing meaningful, feedback driven research in their free time? A lot has changed in the past five years - LLMs being the most important development this century. A world full of high quality, independent science done by hobbyists and part-timers feels more and more possible. So what's stopping us?
Is there an abundance of capable people who want to do independent science? And if so, what's hard about it, and what are the obstacles? And how can I try to overcome them?
This was my journey investigating those questions.
My first big realisation was that there are tons of independent scientists who are already doing great work. There is a large group of people that are interested to start, as well.
The video I made on doing independent science got sixty thousand views. This was more than I expected, and the comments made me realise just how many people are interested in doing research in their free time.
I then made a discord that now has 600 people - that's a lotta people that do science in their free time! People posted their introductions, and I saw cool projects ranging from big datasets in air quality (Aurel Wunsch), to biosecurity in Eastern Europe, to applied physics, to LLM safety, to computational chemistry. A lot of these projects were done in collaboration with mainstream academia. No quacks here!
I realised that there's a bustling group of people that want to do, or are doing independent science alongside their day jobs.
In terms of background, many individuals did research whilst they were in university, but had to give it up as they started their jobs. There were also individuals who worked as engineers in top tech companies, science adjacent companies, or start-up founders, who already had high level experience in project management and research. Some had already submitted papers! Most tried to do research in their free time out of pure passion, and chose day jobs in industry to support financial and lifestyle goals.
I also spoke to 50 people people one-on-one on Google meet, all of whom were independent in science outside of academia. I organised back to back calls on Tuesday and Saturday for two weeks, 20 minutes each.
It was then I learned of some key obstacles stopping more people from doing independent science.
The Obstacles
There are two related problems that seem to stop the independent science community from flourishing.
The main thing I learned from my chats is that independents struggle to find feedback or supervisors for their work. Finding feedback is crucial to doing good work, but there isn't an easy way to get feedback if you're outside of academia.
This happens because
a lot of people currently in academia are too busy to respond to emails from independents. They don't think there is enough upside in supervising someone out of the system.
independent researchers don't know how to approach feedback / supervisors, so have trouble finding research gaps. Even if they do, they don't know a path to build enough background or portfolio to convince a supervisor to work with them.
independent researchers find it hard to manage time effectively with a day job.
there are no public tools that efficiently match independent scientists to supervisors willing to work with them.
We can't do anything about 1!
As for 2, there seem to be two main strategies that people are trying to get feedback: cold emailing and network effects. I made a video explaining how to do this better, but the main idea is that independent researchers need to focus on learning about other people’s research problems first. Solving a problem for someone means that at least one person cares about their work.
Once a problem scope is found, independent researchers then need to come up with a minimal project or value proposition that signals a level of commitment and skill. This email I wrote seemed to work:
Another approach is networking. My biosecurity writing on Substack was enough to get some introductions from folks at key biosecurity non-profits. Going to conferences was a great way to get information on who was willing to offer feedback for my work (thanks to everyone who spoke to me). Once I did that, I wrote out problems that I thought were important, then asked for feedback from 10 people in the field. I've am also working on a global health project, but I only started that when I found a person in the area with an exceptionally high-upside problem to solve.
Once there's a problem to solve, the next step is to iteratively make progress. I've found that a lot of independent researchers struggle to stay motivated, or get started on something. The remedy for this seems to be just taking small steps. This means:
solving a tiiiiiny bit of the problem you are trying to fix
asking for feedback, with cycle times on the order of a few days to a week.
To stay motivated doing this, I've found that recording ones progress, and keeping a healthy task and prioritisation loop helps. I keep an obsidian publish of literally everything that I've thought about. I also reserve my Tuesdays exclusively for calling people or getting feedback.
I found that even with some supervision, independent researchers still struggle to get context, and understand prerequisites needed when reading papers. It seems that the current tools to learn math independently are just not there yet. And most science papers are hard to grasp without context, or relationships with the people who work on that paper.
In an effort to build people's prerequisite knowledge up to research standards, one thing I am trying to do is supervise others to do 1st year undergraduate mathematics, with a tool I've built called Euclia - a way to rapidly offer feedback on higher level math exercises. My aim with this is that in the future, people all around the world can receive supervision to learn high level math.
I am also really excited about creating networks linking researchers in academia to open source contributors. I'm having a livestream with Gabriele Carcassi, the founder of the Assumptions of Physics project later this week on this topic, and I hope to learn more from Gabriele on that.
Acknowledgements
Thank you to Jess Carr and Arthur Wayne for their huge involvement in this project.
This is very interesting, thank you. I was wondering: how do you balance content between your Substack and Obsidian Publish? Do you have a specific strategy or distinct content categories for each platform?
Great article Afiq. "It seems that the current tools to learn math independently are just not there yet." - this is especially true. What I've learned from casually shadowing friends in current math and physics PhD programs is that advanced math, especially the kind that feels like a intuitive tool that you can grasp and weld to accomplish your research goals, is an enormous barrier to entry. For one, it takes many continuous years to master and manipulate at your will. I'm not sure if there will be a way around this besides just having a really good go at your research problems and making all the right choices that lead to you gradually developing that essential familiarity over time.