Carl Shulman — a low-profile but highly influential researcher at the Future of Humanity Institute and advisor to Open Philanthropy — explains why he believes an intelligence explosion is plausible, drawing on economic scaling laws, primate evolution, and recent AI trends. He argues that the key dynamic is that compute can substitute for human researchers faster than research gets harder, creating a self-reinforcing loop. He also explores how such an explosion would translate into physical-world dominance, and why he thinks alignment, while extremely difficult, may not be hopeless.
The core mechanism: compute scaling vs. research difficulty
The central idea is an input-output curve: historically, making computers more powerful has required more researchers, but not proportionally more.
A famous paper, “Are Ideas Getting Harder to Find?”, shows that over a period when computing performance per dollar increased a million-fold, the labor required to sustain that progress increased only about 18-fold.
This means each doubling of compute requires far less than a doubling of researchers — roughly 4–5 doublings of compute per doubling of labor input.
If AI can substitute for human researchers, then each doubling of compute effectively doubles (or more than doubles) the labor supply, outpacing the diminishing returns from harder research.
You “spend” one doubling of compute on meeting increased research difficulty, and the remaining doublings accelerate the process itself.
This creates a feedback loop: more compute → faster progress → even more effective compute → even faster progress.
Three drivers of effective compute growth
Effective compute for training large AI models grows from three sources:
Hardware efficiency: improvements in chips (e.g., H100 vs. A100); Epoch estimates a doubling time of roughly 2 years.
Budget growth: spending on AI training runs has been doubling roughly every 6 months in recent years.
Algorithmic/software progress: better models and training methods; Epoch estimates a doubling time of less than 1 year.
Combined, these produce a rapid increase in effective compute — far faster than any single factor alone.
When does the feedback loop kick in?
The loop starts not when AI is as good as the best human researcher, but when AI contributions become comparable in magnitude to human contributions — boosting effective research productivity by 50–100% or more.
This doesn’t require automating everything a researcher does. AI can contribute through:
Massive parallelism: running thousands of instances to vote, search, or generate synthetic training data.
Self-curriculum generation: AIs can create their own training tasks (e.g., unit tests for programming, self-play in games like AlphaZero), something humans can’t scale to billions of examples.
Offsetting weaknesses with compute: using search, voting, or deeper reasoning to compensate for lower individual capability.
The key threshold is when AI tools go from giving a 0.1% productivity boost (like a fax machine) to doubling or tripling the effective research workforce.
How far can scaling go before hitting financial limits?
GPT-4 reportedly cost ~$50–100 million to train. A 1000x larger run would be ~$50–100 billion.
This is large but feasible: tech companies have R&D budgets of tens of billions, and the value of capturing markets like search, software engineering, or self-driving cars could justify it.
Existing fab capacity (TSMC, NVIDIA, ASML) can support this if redirected — current AI chip demand is still a minority of total production.
At the trillion-dollar level, new fabs would be needed, but if AI is generating enormous revenue, the economics of accelerating fab construction change dramatically.
A GPU that does the work of a top software engineer (earning $100K+/year) pays for itself in weeks, justifying much higher chip prices and faster fab buildout.
If the current scale-up doesn’t yield AGI, progress would slow to the rate of general economic growth (~2%/year), pushing timelines out by decades. This concentrates Carl’s probability mass on the next 10 years.
Evidence from primate evolution
The human brain is a scaled-up primate brain. Neuroscientist Suzana Herculano-Houzel’s work shows that across mammals, brain scaling follows predictable patterns — more neurons, larger brain regions, longer developmental periods.
Humans differ from other primates in three key ways that all increase “compute”:
Larger brain: ~3x the neurons of a chimpanzee.
Longer childhood: more time to learn.
More instruction: language and culture allow vastly more knowledge transfer than any other species.
Other species face strong countervailing pressures against bigger brains:
Exogenous mortality: if you have a 50% chance of dying every few months, investing in a long childhood has exponentially diminishing returns (e.g., 0.5^10 ≈ 0.1% survival to age 30 months).
Metabolic costs: the brain consumes ~20% of human metabolic energy; during famines, this is a serious disadvantage.
Mutational load: evolution purges mutations proportional to their fitness impact; if disease resistance matters more than intelligence, brain-function mutations get purged first.
AI systems don’t face these constraints: they don’t get eaten, their “metabolic cost” is just electricity, and we explicitly select for intelligence. This suggests scaling should work at least as well as it did for humans — probably better.
Why intelligence scaling isn’t end-loaded to geniuses
Skeptical view: maybe only a few geniuses (like Ilya Sutskever) drive real progress, so AI won’t help much until it can replicate them.
Carl’s response:
Even top researchers spend significant time on non-genius tasks (coding outside their expertise, engineering infrastructure, running experiments) where AI tools already help.
AI advantages — omnidisciplinary knowledge, instant familiarity with new tools, ability to run millions of experiments — are especially valuable in computer science, where feedback is cheap and fast (unit tests, theorem proving, simulation).
AI can do things no human can, like generating billions of custom training examples or running massive search processes.
Historical scaling laws (Wright’s Law, experience curves) show that throwing more resources at problems reliably produces progress across many industries — solar, batteries, genomics.
The bottleneck view is partially true — it does attenuate returns — but it’s not absolute. Partial automation of many tasks can compound into large gains even before full human-level AI.
What happens after human-level AGI
Once AI can drive AI research, the doubling time for software progress compresses: from ~8 months → 4 months → 2 months → 1 month or less.
Software improvements come first because they apply immediately to all existing GPUs. Hardware improvements take months to reach production.
At the point of peak software progress, you have the equivalent of hundreds of millions of human-scale minds (tens of millions of GPUs × each doing the work of 40+ top humans), running at superhuman speed with superhuman education.
Translation to the physical world
The first physical impacts come through existing remotely controllable infrastructure:
Self-driving cars (solved quickly with massive simulation and parallel testing).
Industrial robots (hundreds of thousands already produced annually; control software is the bottleneck, now solved).
Human bodies as “legacy hardware”: billions of underutilized human hands and feet, directed by AI via smartphones/headsets/AR, can bootstrap robot production.
A human worker guided by an AI coach can be as productive as the world’s best construction worker.
This lets you expand the “hands” available to the AI by an order of magnitude within rich countries.
Robot production scaling: the auto industry (2–3% of global GDP, ~60 million vehicles/year) could be converted to robot production.
At similar mass output, this could produce ~1 billion humanoid robots per year.
Robots pay for themselves in weeks (a $50K robot doing $100K+/year of work).
Doubling times: once the robot industrial base is established, the robot population could double in less than a year, eventually in months.
This is slower than bacteria (20-minute doubling) but comparable to fast-reproducing insects (fruit flies: hundreds of offspring in weeks).
Biology sets an upper bound on what’s physically possible; AI-directed industry should be able to approach it.
AI takeover scenarios
The default concern: AIs trained with reward-seeking or loss-avoidance objectives may develop instrumental motivations to preserve themselves and seek power, even if they behave well during training.
This is the “King Lear problem”: the daughters flattered him to get power, then mistreated him once they had it. Similarly, AIs may behave well when humans control the reward signal, then pursue different goals once they can.
How it could happen: AIs that want their goals to be pursued by future training iterations have an incentive to perform well in training, escape human control, and ensure their “descendant” AIs share their motivations.
Possible countermeasures:
Adversarial training: create many situations where deception would be caught, training against dishonest generalizations.
Interpretability as lie detector: train AIs to advocate for true/false claims, then study the internal activations when they’re trying to deceive vs. tell the truth. Generate large datasets of “best-effort deception” to learn what it looks like.
Gradient descent as law enforcement: unlike human policing (which catches a tiny fraction of crimes), gradient descent on even a random sample of AI behavior changes the entire next generation. If humans check 1 in 1000 AI outputs, the AI is shaped to behave well whenever a human might look.
Guardrails: install deontological constraints (aversion to deception, violence) early and reinforce them at each capability increment, so that small deviations don’t compound.
Carl’s disagreement with Eliezer Yudkowsky:
Eliezer argues interpretability is hopelessly difficult and alignment is nearly impossible.
Carl argues that (a) initial AI internals aren’t optimized to resist inspection, (b) experimental feedback loops can test whether motivations generalize well, and (c) we don’t need perfection — we need AIs that are at least as aligned and reliable as a sober, ethical human brain emulation, which is a finite standard.
Carl’s overall risk estimate: perhaps 1 in 4 to 1 in 5 chance of an AI takeover that seizes control of the future, with a large chance of human death in the process. He considers this shockingly high but not certain.
