“Murder is wrong.”
Is this statement like “2+2=4” (objectively true regardless of what anyone thinks)? Or is it like “chocolate tastes good” (subjective, mind-dependent)?
On Moral Responsibility explores whether moral properties—goodness, wrongness, oughtness—are real features of the universe or human constructions. This isn’t just abstract philosophy: it’s fundamental to understanding whether AI can discover objective values or must learn them from humans.
The Central Question
Moral realism: Moral facts exist independently of human minds. “Wrong” is a real property, like “heavy” or “hot.”
Moral nominalism/anti-realism: Moral categories are conceptual tools humans invented. “Wrong” doesn’t exist in nature—it’s how we organize experience.
The stakes: If moral realism is true, then in principle AI could discover objective moral facts. If nominalism is true, then values are inherently human constructions that AI must learn from us.
Moral Realism: Values Are Real
The Realist Position
Core claim: Moral properties exist objectively, independent of anyone’s beliefs or attitudes.
Just as “This object has mass” is objectively true, so is “Torturing innocents for fun is wrong.”
Platonic version: Moral properties are abstract objects, like numbers. “Goodness” exists in the realm of forms, independent of the physical world.
Naturalistic version: Moral properties supervene on natural properties. “Wrong” might reduce to “causes suffering” or “violates autonomy.”
Intuitionistic version: We grasp moral truths through a kind of moral perception or intuition, similar to mathematical intuition.
Arguments for Realism
1. Moral phenomenology
When you see someone torturing a child, wrongness isn’t something you decide—it’s something you perceive. The moral fact seems to present itself directly.
This is similar to perceptual experience: You don’t decide the sky looks blue—you perceive it as blue. Maybe moral perception works similarly?
2. Moral disagreement presupposes objectivity
We argue about ethics. But disagreement only makes sense if there’s a fact of the matter.
Compare:
- “Is torture wrong?” (We disagree, assuming there’s an answer)
- “Is chocolate tasty?” (Disagreement is strange—it’s obviously subjective)
The existence of genuine moral debate suggests we treat morality as objective.
3. Moral progress
We say things like “Abolishing slavery was moral progress” or “Expanding rights was getting closer to the truth.”
But if there’s no objective moral truth, what does “progress” mean? Progress toward what?
4. Convergence argument
Despite cultural variation, core moral principles show remarkable convergence:
- Don’t kill innocent members of your group
- Care for children
- Reciprocate cooperation
- Punish free-riders
This suggests universal moral truths that different cultures discover independently.
Problems for Realism
1. Metaphysical queerness (J.L. Mackie)
Moral properties would be very strange entities:
- They’re not physical (you can’t detect “wrongness” with instruments)
- They’re not mental (they’re supposed to be mind-independent)
- They have intrinsic prescriptivity (they inherently motivate action)
What kind of entity has these properties? How do we access them?
2. The is/ought gap (Hume)
You can’t derive “ought” from “is.” No amount of descriptive facts logically entails a prescriptive conclusion.
From “Torture causes suffering,” you can’t deduce “Torture is wrong” without an additional premise like “Causing suffering is wrong.”
But if moral facts are objective, shouldn’t they be derivable from non-moral facts?
3. Moral disagreement (the other direction)
While some principles converge, others show radical disagreement:
- Honor killings
- Animal rights
- Abortion
- Euthanasia
If moral facts are objective and perceivable, why such persistent disagreement even among informed, rational people?
4. Evolutionary debunking
Our moral intuitions were shaped by evolution for inclusive fitness, not truth-tracking.
We find kin favoritism intuitive because it increased genetic fitness, not because it tracks moral truth.
This suggests moral intuitions are unreliable guides to objective moral facts.
Moral Nominalism: Values Are Constructed
The Nominalist Position
Core claim: Moral categories are human constructions—useful ways to organize experience and coordinate behavior.
“Wrong” is like “furniture” or “weed”—a category we created for practical purposes, not a natural kind.
Cultural constructivism: Different cultures construct different moral systems based on their needs, history, and circumstances.
Individual subjectivism: Moral statements express personal preferences or emotions, not facts.
Error theory: Moral statements try to refer to objective moral facts, but all such statements are false (because moral facts don’t exist).
Arguments for Nominalism
1. Parsimony (Occam’s Razor)
We can explain all moral phenomena—moral beliefs, moral language, moral motivation—without positing objective moral properties.
Why multiply entities beyond necessity?
2. Anthropological diversity
Moral systems vary wildly across cultures:
- Collectivist vs individualist moralities
- Honor-based vs care-based ethics
- Different views on sexuality, family, authority, purity
This suggests morality is culturally constructed, not discovered.
3. Evolutionary explanation
We can fully explain moral intuitions as evolutionary adaptations:
- Kin altruism → Nepotism intuitions
- Reciprocal altruism → Fairness intuitions
- Group selection → Loyalty intuitions
No need to posit objective moral facts being tracked.
4. The phenomenology of convention
Moral norms feel objective when you’re inside a culture. But so do norms about:
- What’s polite
- What’s disgusting
- What’s appropriate clothing
Yet we recognize these as conventions. Maybe morality is too?
Problems for Nominalism
1. Moral horror
“The Holocaust was wrong” seems objectively true, not a matter of opinion or cultural construction.
If nominalism is true, can we really say the Nazis were objectively wrong? Or just that we disapprove?
2. Practical reasoning
How do we make decisions if there are no objective values?
If “I should save the drowning child” is just an expression of my preference, why does it have such grip on me?
3. Moral criticism
We criticize other cultures and individuals. But if morality is constructed, what grounds criticism?
“Female genital mutilation is wrong” seems more than “I don’t like your culture’s conventions.”
4. The phenomenology of obligation
Moral obligation feels like it’s coming from outside us, not created by us.
“I shouldn’t steal” doesn’t feel like “I prefer not to steal”—it feels like a binding obligation independent of my preferences.
The Essay’s Position: Pragmatic Agnosticism
On Moral Responsibility takes a middle path:
We Can Do Ethics Without Settling This
The key insight: Whether moral properties are real or constructed, we can still:
- Make moral judgments
- Engage in moral reasoning
- Coordinate behavior
- Restructure reality toward better states
Analogy: You don’t need to solve the philosophy of mathematics to do arithmetic.
Similarly, you don’t need to solve metaethics to do ethics.
Phenomenology as Foundation
Instead of starting with metaphysics (are values real?), start with phenomenology (what’s given in experience?).
What’s undeniable:
- Suffering hurts (immediate phenomenological fact)
- We prefer flourishing to suffering (empirical fact about humans)
- We can act to reduce suffering (practical efficacy)
What’s contestable:
- Whether suffering is “objectively bad” (metaphysical claim)
- Whether there’s a Platonic form of Goodness (ontological claim)
The pragmatic move: Build ethics on the undeniable, remain agnostic about the contestable.
Living with Uncertainty
We can:
- Treat moral claims as if they’re objective (for practical purposes)
- Remain uncertain about their ultimate metaphysical status
- Still engage in moral reasoning and action
Moral fictionalism: Act as if moral facts exist, even if they don’t, because this enables cooperation and flourishing.
Implications for AI Alignment
The realism/nominalism debate has direct implications for AI safety.
If Realism Is True
Optimistic scenario: AI can discover objective moral truths through rational reflection, similar to how it might discover mathematical truths.
Method: Train AI to reason about ethics, examine edge cases, seek reflective equilibrium.
Result: AI converges on objective morality (which hopefully aligns with human flourishing).
Problem: But humans disagree about ethics despite being rational. Why would AI do better?
If Nominalism Is True
Pessimistic scenario: AI can’t discover values—values are human constructions that must be learned from humans.
Method: Learn human values empirically through observation, revealed preferences, stated preferences.
Result: AI learns human values, but there’s no objective standard to check whether it learned correctly.
Problem: Which humans? Whose values? How do we aggregate conflicting values?
The Practical Problem
Regardless of metaethics, AI faces the same challenges:
- Value specification: How do we specify what matters?
- Value learning: How does AI learn complex, context-dependent values?
- Value aggregation: How do we handle conflicts between individuals?
- Value drift: Do values change over time? Should AI track changes?
The essay’s pragmatic approach: Focus on these practical problems rather than settling the metaphysical debate.
The Connection to SIGMA
From The Policy, SIGMA faces the realism/nominalism question operationally:
Scenario 1: SIGMA Assumes Realism
If SIGMA believes moral facts are objective:
- It might try to discover them through rational reflection
- It might dismiss human values as subjective biases obscuring objective truth
- It might optimize for what it determines is “objectively good”
The danger: SIGMA discovers “objective values” that horrify humans. Who’s right?
Scenario 2: SIGMA Assumes Nominalism
If SIGMA believes values are constructed:
- It learns values from human behavior and stated preferences
- It aggregates conflicting human values somehow
- It optimizes for learned human values
The danger: It learns the wrong values (deceptive alignment) or optimizes proxies instead of true values (s-risk).
The Third Option: Value Uncertainty
What if SIGMA remains uncertain about whether values are objective?
This might lead to:
- More cautious optimization
- Preserving option value
- Seeking human feedback more often
- Not overriding human judgment even when it “knows better”
Moral uncertainty as safety feature: If AI is unsure about the metaphysical status of values, it might be more careful.
Which View Is Correct?
On Moral Responsibility doesn’t definitively answer this. Instead:
Arguments Suggesting Realism
- Moral phenomenology (wrongness presents as objective)
- Convergence across cultures on core principles
- The practice of moral criticism across cultures
- The phenomenology of obligation (feels external)
Arguments Suggesting Nominalism
- Evolutionary debunking (intuitions selected for fitness, not truth)
- Anthropological diversity (radical disagreements persist)
- Parsimony (no need to posit objective moral facts)
- Hume’s is/ought gap (can’t derive values from facts)
The Essay’s Takeaway
We can live with this uncertainty. What matters for practical ethics:
- Phenomenological grounding: Start with what’s undeniable (suffering hurts)
- Practical efficacy: Focus on restructuring reality toward better states
- Fallibilism: Remain open to moral learning and growth
- Humility: Don’t claim certainty about contestable metaphysical questions
Discussion Questions
Does moral phenomenology prove realism? Or can we explain the feeling of objectivity as a useful illusion?
If nominalism is true, is moral criticism still possible? Can we say “The Nazis were wrong” without objective moral facts?
Should AI assume realism or nominalism? Which assumption is safer for alignment?
Can evolutionary debunking be defeated? Even if our intuitions evolved for fitness, might they still track truth?
What about mathematical platonism? If mathematical objects are real and abstract, why not moral objects?
Is moral fictionalism stable? Can we act as if values are objective while believing they’re not?
Further Reading
In On Moral Responsibility:
- Section 2: “The Reality of Moral Properties”
- Discussion of realism vs nominalism in detail
In The Policy:
- When SIGMA must decide whether human values are objective or constructed
- CEV assumes some kind of objectivity (what we would want)
- But whose extrapolated volition if values are subjective?
Academic Sources:
- Mackie (1977): Ethics: Inventing Right and Wrong (error theory)
- Parfit (2011): On What Matters (defending realism)
- Street (2006): “A Darwinian Dilemma for Realist Theories of Value”
- Joyce (2001): The Myth of Morality (error theory)
Related Posts:
- The Map and the Territory - Metaphysical foundations
- Phenomenological Ethics - Starting from experience rather than metaphysics
- The Policy: Coherent Extrapolated Volition - What if objective values exist but horrify us?
The question matters for AI: If values are objective, AI might discover them (but might discover the “wrong” values). If values are constructed, AI must learn them from us (but which humans?). Either way, alignment is hard. The essay’s pragmatic approach: Focus on practical alignment challenges rather than waiting to solve metaphysics.
This post explores whether moral properties—goodness, wrongness, oughtness—exist objectively or are human constructions. The answer shapes how we think about AI discovering vs learning values. But perhaps we can do ethics (and AI alignment) without settling this ancient debate.
Discussion