Joseph Carlsmith is a senior research analyst at Open Philanthropy working on existential risk from AI, and a philosophy doctoral student at Oxford. He also runs a blog called Hands and Cities where he writes about philosophy, ethics, and the future. This conversation covers his views on utopian thinking, infinite ethics, anthropics, and the computational requirements for human-level AI.
Utopia and Long-Termism
Carlsmith defines utopia not as a specific endpoint but as a “profoundly better future” — one that is genuinely possible if humanity navigates key risks wisely. He emphasizes that the gap between the present and a truly good future is far larger than most people imagine, more like the difference between being asleep and awake than between a bad job and a beach vacation.
He expects a utopian future to be initially alienating to present-day humans, but argues that if we went through a long process of becoming wiser and more capable — not just solving normative ethics but enhancing our cognitive capacities — we would endorse the result. The path to utopia is more about philosophical and cognitive development than about engineering a specific outcome like “hedonium.”
He pushes back against the idea that utopian thinking is inherently dangerous because historical utopian ideologies led to atrocities. He sees the risk as coming not from believing things can be better, but from rigid, uncooperative action based on that belief. He thinks most modern political visions are actually quite banal and fail to grapple with how much better things could genuinely be.
He draws analogies between utopian thinking and religious or spiritual traditions, noting that both involve extrapolation beyond current experience toward something indescribable. But he stresses that utopia would still be a concrete, finite situation with real frictions and resource constraints — unlike heaven.
He is skeptical of Robin Hanson’s vision of a future economy dominated by digital “ems” (emulated minds) whose wages are driven down to subsistence levels. Carlsmith classifies this as a mediocre or bad outcome, not a utopia, and argues that wise coordination could stave off such competitive pressures.
Human-Level AI and Computational Capacity
Carlsmith authored a report estimating the computational capacity of the human brain, aiming to determine how many floating-point operations per second (FLOPS) would be needed to replicate task-relevant aspects of human cognition. His estimate centers around 10^15 FLOPS, though he emphasizes significant uncertainty.
He used multiple methodologies: analyzing neural mechanisms and signaling complexity, comparing with existing AI systems (especially in vision), examining physical limits on energy consumption per computation, and extrapolating from the brain’s communication capacity. He describes this as a triangulation of weak evidence sources.
There is substantial disagreement among neuroscientists about whether we understand brain signaling well enough to make such estimates. Some emphasize biophysical detail and unknowns, leading to very high compute estimates; others are more confident about mechanistic understanding. Carlsmith leans toward the latter but gives weight to both.
This estimate feeds into work by Ajeya Cotra at Open Philanthropy on AI timelines. Her methodology extrapolates from brain-level compute to training costs using current scaling laws, but introduces major uncertainties — especially “horizon length,” or how many times a system must be run per data point during training. Her distribution of training compute estimates spans from around 10^23 to 10^41 FLOPS.
The approach assumes that transformative AI can be trained without major conceptual breakthroughs relative to current deep learning methods. Carlsmith notes that even if current techniques are suboptimal, algorithmic breakthroughs could make the estimate an overestimate — meaning AI could arrive sooner than the raw scaling extrapolation suggests.
Infinite Ethics
Infinite ethics deals with how we should act and evaluate outcomes in possible infinite worlds. Carlsmith argues this is both theoretically important (many ethical theories break when applied to infinities) and practically relevant (we should have non-zero credence that we live in an infinite universe or can have infinite influence).
Even if our causal influence is physically limited by light speed and entropy, he explores how certain decision theories — particularly evidential decision theory — could give us a form of influence over infinite copies of ourselves in an infinite universe. If your actions are correlated with distant copies, evidential decision theory treats their actions as under your control.
This creates problems for expected value reasoning: even a tiny probability of infinite influence can dominate or break expected utility calculations. Carlsmith argues this is a real issue for long-termism and existential risk, not just a theoretical curiosity.
He discusses the challenge of comparing infinite worlds — for instance, if there are infinite copies doing good and infinite copies doing bad, it’s unclear whether any action changes the total. He acknowledges impossibility results showing that basic ethical constraints cannot all be satisfied simultaneously in infinite contexts.
His practical response is to advocate for survival and increasing wisdom first, deferring the hardest ethical decisions to a future civilization better equipped to handle them. This converges with long-termist priorities around existential risk.
Anthropics: SIA vs. SSA
Carlsmith discusses the debate between two anthropic reasoning frameworks using a thought experiment: you wake up in a white room; God flipped a coin — heads means one person was created, tails means a million. What probability should you assign to heads versus tails?
Self-Sampling Assumption (SSA): You should think it more likely that you’re in a world where people with your evidence are a larger fraction of your reference class. In the coin toss case, since everyone has the same evidence, SSA says stick with 50/50. But in other contexts (like the Doomsday Argument), SSA strongly predicts humanity will go extinct soon, because you’d be a much larger fraction of all humans who ever lived.
Self-Indication Assumption (SIA): You should think it more likely that you exist in worlds with more people in your epistemic situation. In the coin toss case, SIA says the probability of heads is approximately one in a million, because there are a million times more people in the tails world who could be you.
Carlsmith prefers SIA over SSA, arguing that SSA leads to bizarre results: it can imply telekinetic influence (e.g., making a boulder jump by committing to create people if it doesn’t), and it allows you to become confident about the outcome of a fair coin that hasn’t been tossed yet.
SIA has its own problems: it predicts you should be in a universe densely packed with observers (e.g., simulations of you everywhere), which we don’t observe. Carlsmith acknowledges this but argues SIA is still better than SSA. He notes that both frameworks struggle with infinite universes, and sees the breakdown of anthropic reasoning in infinite contexts as analogous to the breakdown of ethics in such contexts.
Moral Scope and Insects
Carlsmith discusses the question of insect moral status, prompted by his own experience sterilizing an ant colony. He has significant uncertainty about the degree to which insects are conscious or can suffer, but thinks it’s a strange view to be extremely confident that insect experiences are totally morally neutral.
He notes that our common-sense reactions (e.g., being disturbed by a child burning ants with a magnifying glass) suggest we already attribute some moral relevance to insects, even if we don’t act on it consistently.
He rejects the reductio ad absurdum that taking insect suffering seriously would require radical lifestyle changes (like never driving). He wants to see the empirical case for what impact we’re actually having and argues there’s a middle ground between ignoring insect suffering entirely and adopting extreme Jain-like practices. The key is recognizing the trade-off and taking responsibility for how you respond, rather than pretending the issue doesn’t exist.
Futurism and the Challenge of Abstraction
Carlsmith observes that much futurism has a “flavor of unreality” — concepts like mind uploads, nanotechnology, and space settlement become so abstracted that they lose their connection to concrete reality. He attributes this partly to the limits of human imagination when trying to model something as vast as the future, and partly to social dynamics where futurism becomes an intellectual game rather than an attempt to describe the real world.
He contrasts this with the concreteness of historical engagement — e.g., imagining the reality of a million men freezing and starving during the retreat from Russia, rather than just tracking border changes. He argues that futurism needs a similar grounding: imagine a concrete scenario, accept it will be wrong, but retain the “flavor of concreteness” even when speaking abstractly.
Blogging and Writing Process
Carlsmith’s blog is a side project alongside his Open Philanthropy work. He maintains high output partly by not over-editing — his posts are long and could be more concise, but condensing them would take more time. He references the aphorism about not having time to write a short letter, so writing a long one.
He started the blog as an exercise in overcoming perfectionism and getting ideas out into the world. He notes that some writers (like Scott Alexander) produce highly readable work through stream-of-consciousness writing, while others (like George Saunders) obsess over every sentence. He sees a trade-off between output volume and polish.
Book Recommendations
The Precipice by Toby Ord — on existential risk, which Carlsmith says conveys the ideas most important to him.
Angels in America (the HBO miniseries or the play) — which he describes as epic and amazing.
Housekeeping by Marilyn Robinson — fiction with a numinous quality.
The work of Nick Bostrom — whose corpus Carlsmith finds valuable and is in dialogue with.
