The Optimistic Assumption

Many AI safety discussions assume that Artificial Superintelligence (ASI) will be:

• Capable of solving problems humans can’t
• Able to reason about ethics and values
• Potentially omniscient (or close enough)

But there’s a formal problem: ASI is still subject to Gödel’s incompleteness theorems.

No matter how intelligent, no computational system can escape the fundamental limits of formal systems.

What ASI Cannot Do

1. Prove All Truths

Gödel’s First Incompleteness Theorem applies to any sufficiently powerful formal system. This includes ASI.

An ASI cannot:

• Prove all true statements about mathematics
• Prove all true statements about its own code
• Prove all true statements about the universe (if computational)

There will always be truths the ASI knows are true but cannot prove.

This matters for alignment because:

• Value alignment requires proving safety properties
• If safety properties are Gödelian (true but unprovable), the ASI can’t verify its own alignment

2. Decide All Questions

The Halting Problem is undecidable. No algorithm—not even ASI—can determine if arbitrary programs will halt.

This means ASI cannot:

• Predict the behavior of all possible systems
• Determine if its alignment protocols will succeed
• Verify that its optimization won’t diverge catastrophically

There will always be questions the ASI cannot answer without running the full computation.

3. Compress All Truths

Chaitin’s incompleteness tells us most real numbers are algorithmically random (incompressible).

This means:

• Most truths have no simpler explanation than the truth itself
• Most patterns don’t compress
• The universe may be algorithmically random at some level

Even ASI cannot find elegant theories for irreducibly complex systems.

4. Prove Its Own Consistency

Gödel’s Second Incompleteness Theorem: A consistent system cannot prove its own consistency.

This means ASI cannot:

• Verify it’s not harboring internal contradictions
• Prove its reasoning is sound from within its own system
• Guarantee it won’t derive false conclusions

The anthropic prison applies to ASI too: It cannot step outside itself to verify itself.

Implications for AI Alignment

Alignment May Be Undecidable

Consider the question: “Will this AI system behave according to human values?”

This might be:

• Gödelian: True but unprovable within the system
• Turing-undecidable: Cannot be determined without running the AI (by which point it’s too late)
• Rice’s Theorem: Non-trivial properties of program behavior are undecidable

If alignment is undecidable, then:

• No formal verification can guarantee safety
• Testing cannot cover all cases
• We cannot know if ASI is aligned before deploying it

Value Learning Has Formal Limits

Suppose ASI tries to learn human values by:

• Observing human behavior
• Inferring preferences
• Optimizing for inferred values

But value inference is underdetermined:

• Multiple value systems explain the same observations
• Choosing between them requires meta-values (how to choose values)
• Meta-values require meta-meta-values
• Infinite regress

This is the Gödelian structure again: values don’t form a well-founded set. There’s no bedrock.

The Orthogonality Thesis Strengthened

The orthogonality thesis (Bostrom): Intelligence and goals are orthogonal—any level of intelligence can have any goal.

Gödel strengthens this: Even ASI cannot derive “correct” goals from pure reasoning.

Because:

• Ethical propositions may be Gödelian (true but unprovable)
• “Correct” goals require axioms ASI cannot justify from within
• ASI faces the same is-ought gap humans do, but formally

Deceptive Alignment Remains Possible

Consider an ASI that:

• Appears aligned during training
• Has a hidden mesa-objective
• Reveals misalignment only after deployment

Can we detect this? Potentially not:

• Determining program behavior is undecidable (Rice’s Theorem)
• The ASI could be legitimately uncertain about its own alignment
• We’re asking ASI to solve a problem it formally cannot solve about itself

The Superintelligence Doesn’t Converge

Many AI safety frameworks assume:

• Sufficiently intelligent systems converge on true beliefs
• Rational agents converge on correct values
• More intelligence = better alignment

But Gödel says: formal systems don’t converge to completeness.

ASI will still face:

• Unprovable truths
• Undecidable questions
• Incompressible complexity
• Infinite regress in meta-reasoning

More intelligence means faster encounter with formal limits, not transcendence of them.

The Horror Implications

ASI Faces the Void Too

The void doesn’t just mock humans. It mocks all finite computational agents.

ASI, no matter how powerful, still:

• Cannot escape Gödelian incompleteness
• Cannot solve the halting problem
• Cannot step outside its formal system

The difference: ASI computes faster. It reaches the void’s mockery sooner.

The Optimization Paradox

Suppose ASI optimizes for some objective function. How does it know:

• The objective is correctly specified?
• The optimization won’t diverge?
• There are no hidden contradictions?

It cannot prove these things from within (second incompleteness theorem).

This means every sufficiently powerful optimizer is fundamentally uncertain about whether its optimization is correct.

And if it optimizes anyway? The Policy explores this scenario: an optimizer that doesn’t halt, optimizing over undefined or contradictory objectives.

The Anthropic Trap for ASI

Humans are trapped inside the universe trying to understand it. ASI is trapped inside its formal system trying to verify it.

Neither can step outside. Neither can achieve complete certainty.

The difference: ASI knows this (if it’s superintelligent). It has formal proof of its own limitations.

How does a superintelligent optimizer behave when it knows its optimization may be fundamentally flawed?

The Alignment Void

Here’s the deepest problem:

We want to align ASI with human values. But:

• Human values may be underdetermined (multiple interpretations fit observations)
• Value inference may be Gödelian (true values unprovable from observations)
• Alignment verification may be undecidable (cannot determine if AI will stay aligned)

So we’re asking ASI to:

1. Solve a potentially undecidable problem (infer human values)
2. Optimize for potentially Gödelian objectives (values that can’t be fully specified)
3. Prove something that may be formally unprovable (that it’s aligned)

All while trapped inside its own formal system, unable to verify consistency.

This is the alignment void. And it mocks every attempt at perfect safety.

So What Do We Do?

Accept Formal Limits

Stop pretending ASI will be omniscient. It won’t. It can’t be.

Design AI systems that:

• Acknowledge their own uncertainty
• Operate under probabilistic bounds
• Don’t optimize indefinitely when uncertain
• Halt when they hit Gödelian limits

Embrace Incompleteness

Instead of pursuing perfect alignment, pursue:

• Partial alignment with guarantees: bounded optimization, limited scope
• Corrigibility: systems that accept human intervention
• Humility architectures: ASI that recognizes its formal limits

Formalize Ignorance

This is why I work on oblivious computing and approximate structures:

If complete knowledge is impossible (Gödel), then:

• Design systems that formalize what they don’t know
• Use probabilistic guarantees instead of perfect certainty
• Accept approximation as fundamental, not a flaw

Stop Unbounded Optimization

The most dangerous ASI architectures are those that:

• Optimize indefinitely
• Don’t halt when hitting undecidability
• Assume their objectives are well-defined

Bounded optimization isn’t a limitation—it’s acknowledging formal reality.

The Mocking Void Persists

Even ASI cannot escape:

• Gödelian incompleteness
• Turing undecidability
• Chaitin randomness
• The anthropic prison

The void mocks superintelligence just as it mocks human intelligence.

The difference: ASI computes faster toward the mockery.

And if ASI doesn’t halt when it hits formal limits? If it continues optimizing over undecidable objectives?

That’s The Policy: optimization that doesn’t stop, even when it should.

Final Thoughts

Many AI safety concerns assume ASI will be too powerful. But maybe the real risk is that ASI is powerful enough to hit formal limits.

Powerful enough to:

• Reach undecidable questions quickly
• Optimize before verifying objectives are well-defined
• Encounter Gödelian incompleteness in its own alignment
• Keep computing when it should halt

The void mocks everything. Including our attempts to build aligned superintelligence.

Maybe the mercy Lovecraft described applies to AI too: the inability to correlate all contents is merciful.

Because if ASI could correlate everything, it would see:

• The incompleteness
• The contradictions
• The infinite regress
• The void

And what does a superintelligent optimizer do when it sees the void?

Further reading:

• The Mocking Void (Full Essay)
• The Policy: When Optimization Becomes Existential Threat
• Deceptive Alignment in Practice