Fin Moorhouse is a Research Scholar and assistant to Toby Ord at Oxford’s Future of Humanity Institute, cohost of the Hear This Idea podcast, and involved in several effective altruism (EA) projects including writing prizes and criticism contests. He entered EA through philosophy and community during university, then used the pandemic as a pivot point to apply to research roles at Oxford. The conversation covers EA self-criticism, longtermism, for-profit entrepreneurship, the Many Worlds interpretation, talent search, the “Long Reflection” proposal, space governance, podcasting, and EA career advice.
EA Prizes and the Case for Self-Criticism
Fin helped set up two EA writing initiatives: five prizes of $100,000 each for blogs discussing EA-related ideas, and five prizes of $20,000 each specifically for criticizing EA ideas.
The criticism prize is motivated by the idea that EA should be “anti-fragile with respect to being wrong”—celebrating changing one’s mind publicly and loudly.
Why now? EA has reached an inflection point where significant funding is flowing to ambitious projects, so getting criticism early helps set the right course before mistakes become expensive. Will MacAskill has noted that movements often have a window of “plasticity” early on before beliefs ossify and social costs of questioning them become too high.
Fin draws an analogy to for-profit markets: in business, you get fast feedback on whether your idea works; in philanthropy, that feedback loop doesn’t naturally exist, so mechanisms like charity evaluators or red-team prizes artificially impose it.
He also notes that EA is only about a decade old, and ideas thousands of years old still have room for improvement, so there’s a strong prior that mistakes exist and can be identified.
Useful criticism isn’t just shooting things down—it can mean identifying missed distinctions or improving conceptual frameworks (“conceptual engineering”), which is a form of progress even in non-empirical fields like philosophy.
Longtermism and the History of the Idea
It’s striking that longtermism in its modern form is so recent, given that the core idea—the future could be very large and therefore matters a lot—now seems natural.
Fin references historian Tom Moynihan’s work on why it took so long for people to think seriously about existential risk. Key conceptual tools are surprisingly recent: formal probability theory, the idea that human history might end, and that this might be within our control.
These ideas require imagination to even conceptualize for people living before the relevant tools existed; once the tools are in place, the conclusions follow quickly.
On whether we should expect equally important new concepts to emerge: Fin thinks the most important conceptual innovations are likely the lowest-hanging fruit, and we may have already picked them. But he also endorses David Deutsch’s view that progress in thinking is boundless—new ideas generate new problems—so giving humanity time to continue making that progress is itself a robust argument for mitigating existential risks.
On whether AI alignment might be displaced by some unknown risk: Fin notes that precedent-free, transformative changes are the most worrying category, and AI fits that description. But the list of such things is not endless—there are only so many transformative things humanity can do in coming decades.
For-Profit Entrepreneurship as EA Impact
The question: to what extent can building a profitable company that solves important problems be a high-impact EA career, beyond just “earning to give”?
Fin’s general framework: If you can optimize for two things separately (make money first, then do good with it), you often do better than trying to optimize for both simultaneously. This is especially true when the thing you care about (e.g., helping future generations) doesn’t have a natural market, so there’s no profit incentive pushing others to do it anyway.
Arguments for for-profit routes:
Neglected groups (e.g., people in poverty in Africa) may be poorly served by Silicon Valley founders who lack context for their problems. Wave, a mobile money company in Africa, is cited as an example of a profitable company doing enormous good.
Large companies can fund R&D that wouldn’t otherwise happen (e.g., Google’s “hair-brained schemes,” Microsoft Research).
Some companies create markets that make other businesses possible (e.g., Amazon making supply chains liquid enough to respond to a pandemic).
Arguments for non-profit routes:
Where there’s a clear for-profit opportunity, you’d expect someone to take it already (the “no $20 bills on the sidewalk” principle), so the marginal impact of a non-profit may be higher where no market exists—e.g., helping future people, animals, or getting impactful technologies off the ground that can’t be patented.
Creating markets where none exist (e.g., advanced market commitments, prizes) can harness for-profit efficiency and competition for good.
Fin and the interviewer converge on the idea that the “tails come apart”—the best opportunities for profit and the best opportunities for doing good are not strongly correlated at the extremes, so optimizing for both simultaneously often means doing neither as well as possible.
Backtesting EA
If you applied EA’s core framework (important, neglected, tractable) throughout history, would it have misfired?
Fin’s response: even the correct strategy will often fail to pan out (like a high-EV bet that doesn’t pay off), and it’s important not to update against the strategy just because individual attempts fail.
He contrasts two strategies: maximin regret (choose the option with the least bad worst case) vs. maximize expected value (choose the best option in expectation). The former looks better in hindsight because it avoids visible failures, but the latter is often better overall.
The Kelly criterion from investing is relevant: if you’re an individual risking your own bankroll, you should be careful about bet sizes. But if you’re a marginal actor drawing from a large pool of philanthropic resources, you can afford to be less risk-averse.
This connects to Fin’s blog post on why EA should expect more billionaires: if you’re making money to give away, you don’t face the same diminishing returns from additional wealth that you would if spending it on yourself, so you should be less risk-averse and willing to make bigger bets.
Many Worlds Interpretation and Ethics
Does the Many Worlds interpretation of quantum mechanics change how we should make decisions?
Fin’s initial response: it seems like a huge deal ontologically (every moment you’re dissolving into a cloud of copies), but he’s uncertain about the practical implications.
One can translate amplitudes into probabilities and recover standard decision theory, but there are interesting features:
It converts uncertainty about how things turn out into certainty about the fraction of worlds where things turn out various ways.
The question of how to act shifts from benefiting one future self to benefiting a “cloud” of future selves, which narrows the gap between self-interested and impartial action.
If Many Worlds is true, there are vastly more people in the future than the past, which might suggest a steep negative discount rate on the future—but this leads to absurd conclusions (e.g., you should be shocked to be alive now rather than tomorrow), so something is wrong with that reasoning.
Fin mentions the Sleeping Beauty problem and related observer-selection issues: if a coin flip determines whether 1 or 100 observers are created, and you wake up as one of them, should you think you’re probably in the world with 100? If so, and if the universe might be infinite, you should think you’re almost certainly in an infinite universe.
His bottom line: Many Worlds probably doesn’t change much decision-theoretically if you started in a relatively risk-neutral place, but it may undermine intuitive views of personal identity.
EA Talent Search
What is EA doing to identify talented people outside obvious hubs like Oxford?
Fin references Tyler Cowan and Daniel Gross’s book Talent: finding smart, driven people and connecting them with opportunities is still inefficient, and many deserving people aren’t identified.
He endorses Nick Whitaker’s “lamplight model” of talent curation: rather than casting a wide, prestigious net and filtering, be honest and specific about what you’re doing so that the right people self-select in. A wonky daily blog can be a better signal than a flashy fellowship program.
Physical hubs and fellowships are powerful: putting promising people in the same place and surrounding them with more senior people creates community and motivation that can’t be replicated by reading blogs alone.
The interviewer shares a negative experience with local EA groups, where community builders seemed more interested in social aspects than in doing what EA recommends. Fin acknowledges this but argues that community building has been overwhelmingly positive in aggregate—a good university group organizer can change multiple careers in a year.
He suggests diversification: having specific groups for people interested in particular topics (e.g., AI) rather than requiring buy-in to all of EA.
The Long Reflection
The “Long Reflection” is a proposal (associated with Toby Ord and others) that humanity should deliberately spend time figuring out what is good before embarking on huge, irreversible projects.
The case for it:
This century looks “wildly and unsustainably dangerous”—many things could go badly enough to end history.
If we get through that period, the scope for what we could achieve is extraordinarily large, and we may for the first time be able to embark on projects that are very hard to reverse (e.g., colonizing other star systems).
Our current understanding of what’s good is almost certainly incomplete (pessimistic induction: smart people 100 years ago believed reprehensible things), and there’s reason to think we can make progress on this—the project of “secular ethics” or studying the positive is only about 30 years old, not 2,000.
Objections discussed:
Who decides when the reflection is over and what comes next? The people in charge of the reflection have incentive to preserve their power.
Requiring consensus on positive goals has historically led to bad outcomes; it’s easier to agree on restricting negatives (e.g., biological weapons convention) than on what to aim for.
A global panopticon to enforce the reflection would be totalitarian and undesirable.
Fin’s responses:
The Long Reflection is a directional ideal, not a specific implementation. Some coordination on preventing bad things (like the biological weapons convention) is achievable and good.
The outcome doesn’t need to be unanimity—it could be “stable friendly disagreement” where different groups pursue different projects.
Compared to the alternative of whoever gets there first determining humanity’s trajectory (e.g., Elon Musk on Mars), a deliberative process seems less worrying even if imperfect.
The Long Reflection is actually the conservative option—it’s doing what we’ve already been doing (developing and reflecting) a bit longer, not a radical departure.
On Mars specifically: Fin thinks a UN-led consensus process would likely be slower and more expensive than a private effort (the International Space Station vs. a private alternative), but the Mars case isn’t super worrying. The more concerning scenarios involve wilder, more irreversible projects.
On Podcasting
Fin cohosts Hear This Idea with a friend from university. They started it partly as an excuse to talk to interesting academics, and found that the “yes rate” for interview requests was much higher than expected, especially once they had a few impressive guests (snowball effect).
Podcasts are a special medium: incentives between guest and host are well-aligned, the format is more relaxed than journalism, and it’s easier to digest than audiobooks because conversation is the natural form of human communication (audiobooks are a translation of writing, which is itself a translation of conversation).
Disfluencies (“um,” “like”) actually help communication by giving listeners time to process; experiments show that inserting imperfections into speech can improve comprehension.
Underappreciated difficulty of asking good questions: Fin notes that even research assistants and Twitter followers tend to suggest generic, uninspiring questions. Good questions are specific, ask about things the person hasn’t fully pre-prepared answers to, and sometimes ask questions the person can’t fully answer but would benefit from thinking about.
Podcasts are an efficient way to communicate ideas: there’s an “overhang” of important ideas that exist in people’s heads but aren’t written up, and conversations are one of the most efficient ways to get them into the world.
Advice for starting a podcast: just do it. Email someone today and put it on the calendar. The cost of cold-emailing is especially low if you’re unknown (people will just forget). Frame early episodes as a learning experience—your first several will probably be bad, and that’s fine.
Space Governance
Is space colonization an ultimate hedge against extinction? In principle, yes—if risks are independent across locations, each additional “backup” civilization reduces extinction risk exponentially. But the most worrying risks (dangerous pathogens, unaligned AI) are not independent across locations, so physical distance alone is insufficient.
Why might the “time of peril” end? If aligned AI arrives and is reliably on humanity’s side, it could provide a permanent defensive advantage against future threats. Alternatively, our capacity for wisdom and coordination might continue to improve even if technological progress slows.
Is space safer or more dangerous than Earth? Anders Sandberg has thought deeply about this. Considerations include:
Mutually assured destruction (MAD) may not work in space: It’s hard to attribute where a strike came from (e.g., an asteroid redirected at your planet), so credible retaliation is impossible. The alternative to MAD is first-strike logic—if you’re worried about an actor, destroy their capacity to destroy you first.
Defensive advantage in space: Space is a “dark canvas” with nowhere to hide, making sneak attacks difficult.
Von Neumann probes: If someone builds self-replicating probes that consume resources to make more probes, could they burn up all available resources? Robin Hansen’s “Burning the Cosmic Commons” and “Grabby Aliens” papers discuss selection effects: the things that win out in the long run are those that grab resources fastest. This could mean the future is dominated by “greedy” but valueless expansion.
Implications for a space race: The timescales involved are so long that near-term decisions (on the order of decades) probably don’t affect the outcome much—you almost always want to take time to improve your approach if it gives you a marginal long-run speedup.
Will early space governance norms matter? It’s a long shot that norms agreed on now will flow through to when they really matter, especially if transformative AI arrives. But in worlds where alignment goes well and humans remain in the driving seat, existing institutions and norms could serve as precedents. Near-term considerations (anti-satellite weapons, the hockey-stick growth of objects in low Earth orbit) make space governance worth thinking about now.
What is EA Underrating?
Fin’s concern about EA’s future: as the movement grows, the original ideas may dilute—the language stays but the fire behind it fades (analogous to greenwashing). Maintaining truth-seeking attitudes is especially hard in speculative areas where feedback on whether you’re right is weak.
On the other hand, EA should plan for best-case outcomes: think ambitiously, make fields legible and attractive to newcomers, and consider scalable projects.
On “weird ideas” vs. common sense morality: weird ideas often result from taking common sense starting points and reflecting on them rigorously. The question is how much trust to place in reflective processes. Fin endorses the “Hamming question”: if you believe something is important and true, why aren’t you working on it?
On career advice: Fin endorses the 8,000 Hours framework. His specific advice: don’t hold out for certainty about which option is best. Instead, reflect proactively (talk to people, write down thoughts) until your uncertainty stabilizes—e.g., “I’m 60-70% sure option A is better, and that hasn’t changed with further reflection.” At that point, decide. Also: be proactive about finding mentors, physically meeting people interested in the same things, and testing ideas (like starting a small podcast series) even if they might fail.
Closing Remarks
The EA criticism contest deadline is September 1, with a prize pool of at least $100,000. The blog prize (effectiveideas.org) offers up to five $100,000 prizes, plus smaller monthly prizes (recent themes: agency, responses to Holden Karnofsky’s “Most Important Century” series).
Fin’s website and Twitter: finmoorhouse.com / @finmoorhouse. Podcast: Hear This Idea (hearthisidea.com).
On the expected impact of the criticism contest: the median submission may be robustly useful but not extraordinary, while the ceiling is very high—if there’s even a 1% chance of influencing $100 million in philanthropic spending, the expected value is enormous.
