A non-technical introduction to AI risk

In the summer of 2008, experts attending the Global Catastrophic Risk Conference assigned a 5% probability to the human species’ going extinct due to “superintelligent AI” by the year 2100. New organizations, like the Centre for the Study of Existential Risk and the Machine Intelligence Research Institute, are springing up to face the challenge of an AI apocalypse. But what is artificial intelligence, and why do people think it’s dangerous?

As it turns out, studying AI risk is useful for gaining a deeper understanding of philosophy of mind and ethics, and a lot of the general theses are accessible to non-experts. So I’ve gathered here a list of short, accessible, informal articles, mostly written by Eliezer Yudkowsky, to serve as a philosophical crash course on the topic. The first half will focus on what makes something intelligent, and what an Artificial General Intelligence is. The second half will focus on what makes such an intelligence ‘friendly‘ — that is, safe and useful — and why this matters.

____________________________________________________________________________

Part I. Building intelligence.

An artificial intelligence is any program or machine that can autonomously and efficiently complete a complex task, like Google Maps, or a xerox machine. One of the largest obstacles to assessing AI risk is overcoming anthropomorphism, the tendency to treat non-humans as though they were quite human-like. Because AIs have complex goals and behaviors, it’s especially difficult not to think of them as people. Having a better understanding of where human intelligence comes from, and how it differs from other complex processes, is an important first step in approaching this challenge with fresh eyes.

1. Power of Intelligence. Why is intelligence important?

2. Ghosts in the Machine. Is building an intelligence from scratch like talking to a person?

3. Artificial Addition. What can we conclude about the nature of intelligence from the fact that we don’t yet understand it?

4. Adaptation-Executers, not Fitness-Maximizers. How do human goals relate to the ‘goals’ of evolution?

5. The Blue-Minimizing Robot. What are the shortcomings of thinking of things as ‘agents’, ‘intelligences’, or ‘optimizers’ with defined values/goals/preferences?

Part II. Intelligence explosion.

Forecasters are worried about Artificial General Intelligence (AGI), an AI that, like a human, can achieve a wide variety of different complex aims. An AGI could think faster than a human, making it better at building new and improved AGI — which would be better still at designing AGI. As this snowballed, AGI would improve itself faster and faster, become increasingly unpredictable and powerful as its design changed. The worry is that we’ll figure out how to make self-improving AGI before we figure out how to safety-proof every link in this chain of AGI-built AGIs.

6. Optimization and the Singularity. What is optimization? As optimization processes, how do evolution, humans, and self-modifying AGI differ?

7. Efficient Cross-Domain Optimization. What is intelligence?

8. The Design Space of Minds-In-General. What else is universally true of intelligences?

9. Plenty of Room Above Us. Why should we expect self-improving AGI to quickly become superintelligent?

Part III. AI risk.

In the Prisoner’s Dilemma, it’s better for both players to cooperate than for both to defect; and we have a natural disdain for human defectors. But an AGI is not a human; it’s just a process that increases its own ability to produce complex, low-probability situations. It doesn’t necessarily experience joy or suffering, doesn’t necessarily possess consciousness or personhood. When we treat it like a human, we not only unduly weight its own artificial ‘preferences’ over real human preferences, but also mistakenly assume that an AGI is motivated by human-like thoughts and emotions. This makes us reliably underestimate the risk involved in engineering an intelligence explosion.

10. The True Prisoner’s Dilemma. What kind of jerk would Defect even knowing the other side Cooperated?

11. Basic AI drives. Why are AGIs dangerous even when they’re indifferent to us?

12. Anthropomorphic Optimism. Why do we think things we hope happen are likelier?

13. The Hidden Complexity of Wishes. How hard is it to directly program an alien intelligence to enact my values?

14. Magical Categories. How hard is it to program an alien intelligence to reconstruct my values from observed patterns?

15. The AI Problem, with Solutions. How hard is it to give AGI predictable values of any sort? More generally, why does AGI risk matter so much?

Part IV. Ends.

A superintelligence has the potential not only to do great harm, but also to greatly benefit humanity. If we want to make sure that whatever AGIs people make respect human values, then we need a better understanding of what those values actually are. Keeping our goals in mind will also make it less likely that we’ll despair of solving the Friendliness problem. The task looks difficult, but we have no way of knowing how hard it will end up being until we’ve invested more resources into safety research. Keeping in mind how much we have to gain, and to lose, advises against both cynicism and complacency.

16. Could Anything Be Right? What do we mean by ‘good’, or ‘valuable’, or ‘moral’?

17. Morality as Fixed Computation. Is it enough to have an AGI improve the fit between my preferences and the world?

18. Serious Stories. What would a true utopia be like?

19. Value is Fragile. If we just sit back and let the universe do its thing, will it still produce value? If we don’t take charge of our future, won’t it still turn out interesting and beautiful on some deeper level?

20. The Gift We Give To Tomorrow. In explaining value, are we explaining it away? Are we making our goals less important?

In conclusion, a summary of the core argument: Five theses, two lemmas, and a couple of strategic implications.

____________________________________________________________________________

If you’re convinced, MIRI has put together a list of ways you can get involved in promoting AI safety research. You can also share this post and start conversations about it, to put the issue on more people’s radars. If you want to read on, check out the more in-depth articles below.

____________________________________________________________________________

Further reading

God is no libertarian

So the world was made by a perfectly benevolent, compassionate, loving God. Yet suffering exists.

Why would a nice guy like God make a world filled with so much nastiness? All these wars, diseases, ichneumon wasps—what possible good purpose could they all serve?

We want God to make our lives meaningful, purpose-driven. Yet we don’t want that purpose to be super depressing. ‘God is a nice guy from his own perspective, but a total asshole by all human standards’ would be a pretty unsatisfying theodicy, and a terrible way to fill the pews. So how do we square a good God with a wicked world?

The standard response is that being truly good requires that one love freedom. God is so good that he won’t interfere with human freedom by preventing suffering. That certainly sounds nice; we don’t want to make autocracy the highest good. But how can this work in practice?

The idea seems to be that we are somehow to blame for our suffering. God, then, is off the hook. We’re free to blame ourselves (rather than God) for whatever evil things befall us. What’s more, we’re free to credit God (rather than ourselves) for whatever good things we accomplish. In this way we can, if we wish, preserve the pure wretchedness of man and the pure excellence of God. We are free to translate the complexity of human experience into a crisp conflict between total sin and total virtue. But there are deep problems with this approach: The shape of our world seems profoundly unlike the shape we’d expect from a libertarian architect.

____________________________________________________________________

First Problem: Natural evil limits freedom.

It’s clear that not all suffering stems from human action. If God had protected the 230,000+ victims of the 2004 tsunami, how would this have interfered with human freedom? Would it not, if anything, it have increased our freedom, by giving the tsunami’s victims a chance to live out their lives?

One might respond that the tsunami’s destruction could have been greatly reduced by human actions. Perhaps God gave us just enough power to save ourselves, and we simply did not employ it.

But blaming the victims simply does not work here. No matter what we had done, we could not have saved every life. And if some people were to blame for the level of devastation, surely those people should have been punished, not innocent bystanders. Which brings us to…

Second Problem: Human evil limits freedom.

If God loves freedom, why does he let people obstruct and enslave one another? Why does he allow oppressors more freedoms than the oppressed? Why not give us just enough freedom to control our own lives, so long as it does not infringe upon the freedom of others?

The problem of evil raises special concerns for individual freedom. You might claim, for example, that humans (and not God) are responsible even for natural disasters, because Adam and Eve introduced suffering and death into the world when they disobeyed God. But that is not a crime committed by every human being, such that every human deserves punishment for it. It is a crime committed by two particular humans. How can we justify punishing someone else for a perfect stranger’s crime? Certainly it is not my fault if I was born to a sinful father. We can’t choose our parents.

(To my knowledge, Origen is the only theologian to have ever resolved this problem. Unfortunately, later thinkers generally consider Origen’s views heretical, and even Origen falls victim to religion’s standard “blame the victim” mentality.)

Third Problem: Our freedom is physically limited.

It’s easy to say that God loves freedom by counting the hits (look at all the things he lets us do!) and ignoring the misses (the things we can’t do). But of course we aren’t free to do whatever we want. God created us in a very specific way, strictly limiting what we can will ourselves to do. We can’t fly merely by flapping our arms. We can’t will aches and pains to go away. We can’t even go directly to Heaven merely by willing it.

So what? What’s the problem? Well, we’ve granted that freedom isn’t absolute, that a good God would make beings free in some respects, but not in others. But now we are forced to explain why God limits our freedom in the particular way that he does. Why give us the freedom to make sandwiches and fire guns, but not the freedom to cure all diseases or teleport away from natural disasters?

If we can’t even begin to explain this, then ‘God loves freedom’ ceases to be a viable justification for suffering. The question is now why God loves this particular freedom (the ‘freedom’ to suffer even when we’d prefer not to) more than he loves rival freedoms (the freedom not to suffer!). A generic appeal to ‘freedom’ can’t even begin to address this question.

Fourth Problem: Our freedom is epistemically limited.

This is the problem of ignorance, a far deeper and thornier issue than the standard problem of evil. What can it mean to say that God respects freedom, when he obviously doesn’t respect informed freedom?

Freedom, in fact, seems quite meaningless when it is not informed. Imagine a child told to pick between two closed doors. Behind one door is a fierce tiger, and behind the other door is chocolate. If the child chooses the door that happens to have a tiger, can we blame the child for his messy death? Surely not.

Yet we, too, live in a world we scarcely understand. It is often claimed that God hides himself from us in order to give us the freedom to doubt him, to choose our beliefs for ourselves. But in fact God’s hiddenness has the opposite effect; it takes away from us our freedom—our freedom to make an informed choice. Since we do not know which religion, if any, is the correct one, we can hardly be blamed if we err. Yet theists assert that those who fail to find God will suffer (e.g., in Hell or merely ‘the absence of God’), and that they deserve to suffer.

Being forced to play Russian roulette, and then losing, is not the same as committing suicide. The freedom to guess is not freedom. It’s just a slavery to chance. Only the freedom to choose between options whose consequences we fully comprehend is genuine freedom, because only then do we really know what option we’re choosing. Yet clearly God did not create beings who fully understand their actions’ consequences. Least of all in the realm of religion.

____________________________________________________________________

The notion of ‘freedom’ favored by our allegedly well-meaning deity, then, ends up looking extremely peculiar. God evidently only loves freedom when it can infringe upon (and be infringed upon by) others’ freedom, and when it is severely limited in seemingly arbitrary ways, such that we are not free to escape suffering in this life or to make informed choices. After qualifying what God prizes in so many strange ways, what evidence remains for the supposition that these preferences even slightly resemble what we call “morality” or “compassion” in the case of humans?

I can think of four possible responses.

  • To Problem 1: Perhaps God created a perfectly orderly world, and in such a world it was inevitable that some disasters would arise.

This doesn’t explain why God created the particular world he did, or why he created at all. It also doesn’t explain why God prizes abstract “order” more than human welfare. Couldn’t he create a world that naturally has typhoons, yet still intervene to save the people victimized by his natural order? The fact that buildings inevitably fall down sometimes doesn’t make it any less immoral to choose not to save people from falling buildings if you’re able.

  • To Problem 2: It’s not God’s fault that humans hurt one another.

The issue isn’t that God’s to blame for everything humans do. It’s that God chose to limit human freedom in one way, but not in another. He made us free to harm one another, but not free to be safe from others’ harm. What makes the former freedom more important than the latter? Why is the villain’s freedom prized above the victim’s? Even if God doesn’t directly cause every human action, he still chose which possibilities to leave open. That calls for explanation.

  • To Problem 3: If we could do anything, we’d be God.

This relies on a false dilemma. It’s not that case that God needs to either make humans omnipotent, or deny them the ability to escape suffering. He could easily give them that one ability, while continuing to deny them other abilities. This on its own would radically decrease the suffering in the world, and radically increase people’s freedom.

And, as an aside: What’s wrong with being God? God sure seems to like it!

  • To Problem 4: If we knew everything, we’d be God.

Again, we don’t need to be omniscient merely to know the consequences of our actions. God chose to create beings that are ignorant of almost everything. If such beings sin without fully understanding the consequences, they cannot ethically be held more responsible than God for what ensues.

Harry Potter and the Fuzzies of Altruism

This is a shorter version of a post for Miri Mogilevsky’s blog, Brute Reason.

Effective Altruists are do-gooders with a special interest in researching the very best ways to do good, such as high-impact poverty reduction and existential risk reduction. A surprising number of them are also Harry Potter fans, probably owing to the success of the EA-promoting fanfic Harry Potter and the Methods of Rationality.

The author, Eliezer Yudkowsky, calls that nice inner glow you feel when you help people “warm fuzzies“. But it’s a common error to assume that everyone thinks and perceives the same way you do, and I’ve come to notice that not everyone who’s interested in charity and social justice gets identical “fuzzies”. People with the same humanitarian goals can differ, not just in their philosophy and tactics but even in their basic psychological motivations. So I decided to construct a taxonomy of fuzzies modeled after the four Houses of Hogwarts.

____________________________________________________________________

slytherfuzzies — how it feels to save the world by improving yourself, mastering your own will, and achieving your personal goals. Slytherfuzzies are that self-esteem boost, that sense of being effective and just plain Awesome, when you successfully help people. At an extreme, people’s happiness is seen as a tool for achieving slytherfuzzies (or just Victory), rather than your drives being a tool to help others. Picture Gandhi cackling in a darkened, smoke-filled room, muttering, ‘All goes according to plan…’

____________________________________________________________________

ravenfuzzies — how it feels to save the world as an intellectually stimulating puzzle. One helps people not so much out of felt empathy as out of boredom, or curiosity, or a conviction that happy, healthy human-style intelligences help make the world a more beautiful, interesting, and complicated place.

____________________________________________________________________

gryffinfuzzies — how it feels to save the world from within a hero narrative, (e)utopian vision, or any sort of Moral Quest. A gryffinfuzzy can be as proud as a slytherfuzzy, but the grounds for pride are externalized — things are finally The Right Way, not necessarily my right way.

____________________________________________________________________

hufflefuzzies — how it feels to save the world in the form of lots and lots of sick baby bunnies. Hufflefuzzies are warm. Personal. Social. Fuzzy. They’re probably the most common and essential source of altruism. They are units of reverse schadenfreude, of empathic joy, of emotional connection, solidarity, or belonging.

____________________________________________________________________

In my own case, I seem to be mostly motivated by gryffinfuzzies. I find that surprising, because philosophically I’m much more likely to explain and defend my ethical views in terms of the value of empathy (like a hufflepuff bodhisattva), or the value of diversity (like a ravenclaw Feyerabendian), or just in terms of my personal preferences (like a slytherin existentialist). Apparently my core moral intuitions are quite distinct from my intellectualizations of morality.

What about you? What drives you to do good? What combinations of fuzzies do you experience, or would you like to? Do they vary for different kinds of charitable work? Do my groupings make sense to you, and are there any fuzzies I’ve left out?

Moral theory is for moral practice

Sam Harris has argued that we should treat situations as morally desirable in proportion to their share of experiential well-being. In a debate, William Lane Craig objected:

On the next-to-last page of his book, Dr. Harris makes the telling admission that if people like rapists, liars, and thieves could be just as happy as good people, then his “moral landscape” would no longer be a moral landscape. Rather, it would just be a continuum of well-being whose peaks are occupied by good and bad people, or evil people, alike. […] The peaks of well-being could be occupied by evil people. But that entails that in the actual world, the continuum of well-being and the moral landscape are not identical either. For identity is a necessary relation.

I think the real problem here isn’t that it could be moral to make evil people happy. Harris and I gladly bite that bullet. The deeper worry is that, in a world teeming with pathological sadists, torturing a minority might well increase aggregate psychological welfare. Yet it would be absurd to conclude that torturing an innocent in such a world is moral.

This is a perfectly fair argument. But Harris simply responds, “Not a realistic concern.

Why the lack of interest? Because, I think, any claim that the English-language word ‘good’ means ‘well-being’, picking it out across all possible worlds, is beside the point for Harris.

A world of sociopaths or sadists would be trapped in a valley of the moral landscape. Fixating on a few tiny hills at the bottom of that valley is missing the big picture, which is that the truly moral act would be to cure the world of its antisocial tendencies, not to indulge them. It’s sort of ‘moral’ for a doctor to spend most of her time making delicious pies for her rapidly deteriorating patients. I mean, baking for others is a good deed, right? But it’s immoral on a deeper level if it distracts the doctor from diagnosing or treating her patients. Craig’s example is alien enough to do some violence to an exact identification of ‘good’ with ‘well-being’, but it does nothing to undermine the enterprise of improving psychological welfare, because it misses the landscape for the hills in much the way the baker-doctor does.

So what is  Harris’ goal in The Moral Landscape ? He seems to want to establish four main theses:

1. Positive experience is what we value.
All the things we care about are instances of experiential well-being.

2. So we should value all positive experience.
Our strongest unreflective desires will be furthered if we come to value such experience in general, however and wherever it manifests. For this binds all of our values together, encouraging us to work together on satisfying them.

3. Morality is about satisfying that universal value.
Since this is the most inclusive normative project we could all legitimately collaborate on, and since it overlaps a great deal with our most rationally defensible moral intuitions, it makes consummate sense to call this project ‘morality’.

4. So science is essential for getting morality right.
The best way to fulfill this valuing-of-experienced-value is to empirically study the conditions for strongly valenced experience.

I’m very skeptical about 1 on any strong interpretation, but I’ll talk about that another time. (EDIT: See Loving the merely physical.) Though Harris places a lot of emphasis on 1, I don’t think it is needed to affirm 2, 3, or 4. Suppose we learn that some people really do value living outside the Matrix, keeping natural wonders intact, promoting ‘purity‘, obeying Yahweh, or doing the right thing for its own sake, and not solely the possible experiential effects of those things. Still Harris could argue that, say…

  • … those goals form a much less consistent whole than do the experiential ones. Perhaps, for instance, subjective projects come into conflict less often than objective ones because we have separate mental lives, but only one shared physical world.
  • education or philosophical reflection tends to make those goals less appealing.
  • … those goals make dubious metaphysical assumptions, in a way experiential goals don’t.
  • … those goals depend for their justification on experiential ones.
  • … those goals causally depend on experiential ones.
  • … those goals are somehow defective variants on, or limiting cases of, experiential ones.
  • … those goals are unusually rare, unusually temporally unstable, or unusually low-intensity.
  • … those goals are so different from experiential ones that they can’t all reasonably be lumped into a single category.

Some combination of the above conclusions could establish that experience-centered goals form a natural group that should, for pragmatic or theoretical reasons, be discussed in isolation. Once we’ve got such a group, we can then argue that our most prized goals will be furthered if we generically endorse the entire category (2), and that these goals will be further furthered if we reserve ethical language for this category (3). 4 will then fall out of 2 and 3 easily, as an empirical conclusion about the usefulness of empiricism itself.

On my view, then, the real action is in the case for 2 and 3. What is that case?

Why value value?

It’s important to highlight here that Harris doesn’t think everyone already generically values all positive experience. It would be a fallacy to deduce ‘everyone values every positive experience’ from ‘everything that’s valued by anyone is a positive experience’.

[I]n the moral sphere, it is safe to begin with the premise that it is good to avoid behaving in such a way as to produce the worst possible misery for everyone. I am not claiming that most of us personally care about the experience of all conscious beings; I am saying that a universe in which all conscious beings suffer the worst possible misery is worse than a universe in which they experience well-being. This is all we need to speak about “moral truth” in the context of science.

So Harris is proposing that we change our priorities. They should change in pretty much the same way our ancestors’ linguistic, political, and intellectual practices changed to affirm the scientific character and universal value of health.

Why change? Because it will allow us to better collaborate on the things we already care about most. Again, why should we prize health in general, as opposed to caring specifically about the health of certain groups of people, or certain body parts? Why not have medicine focus disproportionately on our right legs, disregarding our left legs almost completely? Well, I suppose there are no unconditional, metaphysically fundamental reasons to value health in general, or to build sciences and social institutions dedicated to understanding and improving it. But it’s simpler that way, and it benefits us both individually and collectively, so… why not?

Valuing every experienced value, in proportion to its intensity and frequency, is egalitarian in spirit. Practically democratic. That doesn’t make it ‘objective’ in any mysterious cosmic sense. But it does make it an extraordinarily useful Schelling point, a slightly arbitrary but stable and fair-minded convention for resolving disputes.

Of course, if we just think of it as an arbitrary convention, without ascribing it any importance — if we ‘mere‘ it — then the whole point of the convention will be lost. If no one had any respect for democracy, democracy would dissolve overnight. It may be very important for the practice of valuing value that we adopt moral realism or consequentialism as an absolute law, even if the justification for doing so isn’t so much philosophical first principles or linguistic definitions as our lived, pragmatic concern for our own and others’ actual welfare. Good conventions save lives.

It’s because we do in fact have conflicting desires that it’s important to have a general framework for resolving disputes, and Harris’ is a surprisingly flexible yet sturdy one. On Harris’ view, we do factor values like nepotism and egoism into our calculus, and try to help even sociopaths live a joyful, fulfilling, beautiful life — within limits.

What limits? Simply that it come at no cost to everyone’s joy, fulfillment, and beauty. In that respect, the system is more fair than a democracy, since unpopular values get equal weight; and at the same time less exploitable than one, since that weight is determined by psychological fact, not by popular opinion.

So most malign values are quelched or stymied not because they’re intrinsically Evil but because they don’t scale well. They don’t interact in such a way that they form sustainable ecosystems of positively valenced experience. On Harris’ view, we shouldn’t block or assist sadists and war criminals merely because it pre-reflectively ‘feels righteous’ to do so; for our sense of righteousness can go horribly astray. Rather, we should do so because an ecumenical ‘value all values’ project demands it, and because abandoning this meta-value means abandoning our best hope for fully general cooperation between sentients.

What’s on the table is less a moral theory than a humanitarian superproject. Harris reinterprets our language of ‘ought’ and ‘should’ not with the goal of solving Kantian paradoxes but with the goal of defining and motivating a long-term civilizational research program, all while bringing our intellectual drives and traditions into a more intimate conversation with our moral drives and traditions, at the individual as well as the societal scale.

Why call this ‘morality’?

For a person who wrote a book about meta-ethics, Harris is remarkably unconcerned with meta-ethics. He takes note of it only to do a bit of conceptual and rhetorical tidying up. At all times, his sights remain firmly fixed on applied ethics, on politics, on, well, real life.

[T]he fact that millions of people use the term “morality” as a synonym for religious dogmatism, racism, sexism, or other failures of insight and compassion should not oblige us to merely accept their terminology until the end of time.

But if there’s real disagreement here, why speak in terms of ‘ought’ and ‘bad’ at all?

The problem isn’t that those are univocal, clearly-defined terms whose entrenched meanings Harris is flouting. The more realistic worry, rather, is that they’re horribly confused terms with only a limited amount of consistency within and across linguistic communities. Folk morality is a mess. Heck, academic morality is a mess. And folk meta-ethics and folk normative ethics (and their academic counterparts) are particularly confused and divergent — far more so than object-level morality. So if Harris’ goal is to inject some clarity and points of basic consensus into this conceptual cacophony, why enter the fray we call ‘ethics’, with its centuries of accumulated obscurity, at all? Why not just invent a new set of terms for what he has in mind, like ‘flought’ and ‘flad’? Then, stipulatively, we could have our flobligation cake and eat it too. If he did that, you can be sure that you’d see fewer people treating ‘but you’re just defining morality as “the maximization of well-being”‘ as an objection.

Although it’s tempting to reboot ethics and start over with a clean slate, I think that the risks should we completely forsake the moral conversation are too dire. Moral language is just a language. (What’s ethical remains ethical, whether we call it ‘ethical’ or ‘flethical’, or ‘unethical’, or ‘linoleum’.) But language matters. Our intuitions are language-shaped. Even if we say that ‘florality’ or ‘neuro-eudaimonics‘ is far more humanly important and conceptually deep than traditional ‘morality’, people raised on the ‘morality’ lexicon will still reliably misconstrue how high the stakes are, misconstrue even their own preferences, if we toss out moral language.

Many [highly educated men and women …] claim that a scientific foundation for morality would serve no purpose in any case. They think we can combat human evil all the while knowing that our notions of “good” and “evil” are completely unwarranted. It is always amusing when the same people then hesitate to condemn specific instances of patently abominable behavior. I don’t think one has fully enjoyed the life of the mind until one has seen a celebrated scholar defend the “contextual” legitimacy of the burqa, or of female genital mutilation, a mere thirty seconds after announcing that mortal relativism does nothing to diminish a person’s commitment to making the world a better place.

Moreover, our traditional talk of goodness and badness has some very useful features, like its correlation with our deepest concerns and its built-in universality. Certainly we could redefine morality in, say, egoist terms. ‘Justice’ and ‘ought’ could be made to refer to the speaker’s interests, as opposed to the overall interests of sentient beings. But then it would be less useful as a language, since the meanings of the terms would vary from person to person, like pronouns do, and since we already have adequate ways to express personal preferences.

Ethical discourse is our only established way to concisely refer to aggregate preference satisfaction. So streamlining the expression-conditions of this discourse, stripping it of the parochial or metaphysically dubious associations it has in certain linguistic communities, may be a very valuable project if we have a sufficiently important candidate meaning to adopt. Harris thinks that psychological well-being meets that condition.

I’ve emphasized the revisionary nature of Harris’ project, because I want to make it clear why objections like Craig’s are beside the point. Harris’ goal is to provide a framework for thinking and talking clearly about humanity’s most important (i.e., most widely and deeply valued) problems and possibilities. His goal isn’t to provide a novel theory that can ground all our naïve normative intuitions, ordinary prescriptive language, or sophisticated ethical theories, because he thinks that all three of these are frequently useless, internally inconsistent, even outright contentless.

Everyone has an intuitive “physics,” but much of our intuitive physics is wrong (with respect to the goal of describing the behavior of matter). Only physicists have a deep understanding of the laws that govern the behavior of matter in our universe. I am arguing that everyone also has an intuitive “morality,” but much of our intuitive morality is clearly wrong (with respect to the goal of maximizing personal and collective well-being).

At the same time, I don’t want to suggest that Harris’ framework is all that ethically novel or strange. We really do care with unparalleled ferocity about suffering, rapture, beauty, tranquility, and all the other qualities of experience Harris is interested in. And our everyday moral intuitions and conventions really do orbit the distribution of extreme forms of these experiences.

My qualification is that that’s a contingent fact, and it’s not the core reason Harris is so interested in this project. If our moral intuitions had turned out to be consistently detrimental to our psychological welfare, Harris would have advocated the destruction of morality, not its reconceptualization! But, for all that, the conservatism of Harris’ proposal is very much worth keeping in mind. If nothing else, it shows that Harris’ project isn’t as difficult as it might seem. All we need is a small but vocal pool of intellectuals and public figures on our side, just large enough to reverse the current cultural trend towards blind relativism and lame nihilism.

Harris’ aim, then, isn’t to give a fully general semantic theory of what the word ‘good’ means in English, or to provide metaphysical truth-conditions for all our intuitive judgments. It’s to recommend a simple framework for collaborating on issues of deep humanistic import. It’s to repurpose an increasingly unproductive discourse to express the urgency of scientifically inquiring into the nature of anything and everything that matters to us. And then actually doing something about it.

Regimenting our concept of “morality” with simplicity will make it easy to teach and explain the value of value, regimenting it with elegance will make it easy to theoretically and pragmatically defend the value of value, and regimenting it with egalitarianism will ensure that we do not disregard any of the core concerns of any of the beings capable of having concerns. If Harris’ own proposal is not ideal for this aim, still it seems clear that something has to fill the void that is modern ethical thought, lest this void continue to encroach upon the things we love.

Further reading:

What is a self?

This is a revised version of an IU Philosophical Society blog post.

At the Philosophical Society’s first spring meeting, I opened with a methodological point: Semantics matters. Misunderstanding is everywhere, and it is dangerous. If we don’t clarify what we mean, then we’ll never pinpoint where exactly we non-verbally disagree.

But the importance of semantics doesn’t mean we should fetishize which particular words we use. Just the opposite: In analyzing what we mean, we frequently discover that the world doesn’t neatly break down into the shape of our linguistic categories. We may have one word (“monkey”) where there are really two or three things, or two words (“electricity” and “magnetism”) that pick out the same phenomenon in different guises. Thus we talked about the value of “Tabooing your words”, of trying to find paraphrases and concrete examples for terms whose meaning is unclear or under dispute.

This is of special relevance to discussions of the self. People mean a lot of different things by “self”. Even if in the end those things turn out to be perfectly correlated or outright identical, we need to begin by carefully distinguishing them so that we can ask about their relatedness without prejudging it.

For example: DavidPerry noted that many classical Buddhist texts denied the existence of a self. But what they actually denied was what they called ātman, which some people have translated as “self”. Even had they written in English, for that matter, it wouldn’t necessarily have been obvious which ideas of “self” they had in mind — and, importantly, which they didn’t have in mind.

What are some of the concepts of “self” that we came up with? I lumped them into five broad categories.

1. Thing

EMpyloriWhen we say “That’s an ugly coat of paint, but I like the house itself,” we don’t have the same thing in mind as when we say “I have a self”. It may seem trivial to note that objects in general can themselves be called “selves”; but this has real relevance, for example, to the Buddhist critique of “self”, which really does generalize to all objects — for early Buddhists, humans lack a “self” for much the same reason chariots lack a “self”, because they aren’t things in quite the way we normally take them to be.

Some things, of course, may be more intuitively “selfy” than others. The idea of discrete organic selves, or organisms, is applied to everything from viruses to humans. In this biological sense, I am my body, even though my body can change drastically over time.

Two troubling questions arise here, and they’ll recur for our other ideas of “self”. First, can my concept of myself as an organism be trumped by other (say, more psychological) conceptions? If not, then if my brain were turned into a sentient machine, or if my body perished while my soul lived on, I would not survive! Some ghostly or robotic impersonator would survive, while the “real me” perished with my body. Could that be right? Or is the “real me” something more abstract? And why does the question of which “me” is “real” feel like it matters so much?

2. Persona

By “self” or “person” we sometimes mean the specific things that make you who you are. We mean someone’s personality, character, life-experiences, social roles, and so on. As the Stanford Encyclopedia article on selfhood notes:

We often speak of one’s “personal identity” as what makes one the person one is. Your identity in this sense consists roughly of what makes you unique as an individual and different from others. Or it is the way you see or define yourself, or the network of values and convictions that structure your life. This individual identity is a property (or set of properties). Presumably it is one you have only contingently: you might have had a different identity from the one you in fact have. It is also a property that you may have only temporarily: you could swap your current individual identity for a new one, or perhaps even get by without any.

3. Subject

png;base646bdd8702569ffd85We may also have a more generic idea in mind — a “self” as a subject of experience. But this too conflates several ideas.

First, there’s the idea of an experiencing subject, an experiencer. At a minimum, this could be whatever directly brings experiences about. But does this causal notion adequately incorporate our intuition of a self that “undergoes” or “has” its experiences? What would we have to add to turn an experience-generator into an unconscious self? And if some brain region or ectoglob can be “me”, where do we draw the line between the parts of the world that are me and the parts that aren’t?

Jonathon, for one, voiced skepticism about there being any fact of the matter about the dividing line between Me and Everything Else. Some philosophers even reject the very idea that a self exists “outside” or “behind” experience:

But even so, there remains the distinct idea of an experienced subject. Our self isn’t just hidden behind our experiences; it’s also indicated within them. Thus we can speak of experiences that are “self-aware”, in different ways and to different extents. This ranges from the self-awareness of explicit thoughts like “I am getting rained on!” to primitive perceptual impressions that a certain hand is Me while a certain chair is Not Me.

At the outer edge of this category, DavidPerry raised the idea of a bare “phenomenological” subject, which I took to be the perspectivalness or subject-object structure in experience. Here our discussion became very murky, and DavidBeard expressed some skepticism about the possibility of disentangling this idea from the very idea of consciousness.

In general, we had a number of difficulties reconciling the philosophical method of phenomenology, or describing how things appear from a first-person perspective, with the method of third-person science. Most basically, Neeraj asked, can the fact of first-person experience itself be accounted for in objective, scientific terms? As Briénne put it: Supposing I were an intelligent zombie or automaton, could you explain to me what this thing you call “consciousness” is? This brought us to another way of conceiving a self — behaviorally.

4. Agent

png;base64f4b49a5756f7bece“Self” can be defined in behavioral terms. We generally say that humans and animals can perform actions and deeds, while beaches and kaleidoscopes, metaphors aside, cannot. So agency is an important way of distinguishing persons from non-persons.

Of course, “action” is a vague category. It’s easiest to tell persons from non-persons apart when we’re dealing with intelligent agency, i.e., behaving in a skillful, adaptive, goal-oriented way. We debated whether intelligent behavior can occur in the absence of conscious thought, and if so how we could ever identify subjects of experience based on how they act. Sam noted that we very readily ascribe agency, and perhaps even awareness, to beings based merely on their superficial resemblance to humans and other animals — suggesting that our agent-detecting intuitions are prone to leading us astray.

We might also distinguish deliberative agency, which makes decisions, from rudimentary animal behaviors that possibly lack real “choice.” Even more narrowly, we can ask what gives deliberative agents (or agents in general) free agency. Does social or political freedom, as Nathaniel suggested, inform our concept of “person”? Does psychological or metaphysical freedom help determine whether something is a self in the first place?

This brought to the forefront the important fact that our idea of “self” is not merely descriptive; it is also prescriptive. What things we call “person” is bound up with our values, preferences, and principles. Thus we have to ask how the above ideas relate to moral agency, a being’s responsibility for its own actions. A storm can make bad things happen, but it’s not the storm’s fault. What sorts of things can be at fault?

5. Patient

png;base643f39801d7f0875daJust as an agent is something that acts, a patient is something that’s acted upon. Thus, along similar lines, we can ask what beings are moral patients — beings that can be harmed or benefitted. And we can ask whether there is a special, narrower category of personal patients — whether, for example, humans or intelligent agents have their own special rights above and beyond those of other sentient beings.

But the normative concepts of self aren’t just about morality. We also need to know what it takes to count as a prudential patient. Or, to ditch the jargon: What does it take for something to be the proper object of my self-interest? What sorts of things can be me, when it comes to my looking out for my own welfare?

The question seems so basic as to be bizarre. But in fact it’s not a trivial matter to figure out why I should care about myself — or, given that I do care about myself, what it takes for a thing to qualify as “me” — or how to go about discovering which things so qualify!

More generally, we can distinguish two questions:

1. What does it take to be a certain kind of self? What makes Bob, say, an agent?

2. What does it take for two things to be the same particular self? What makes Bob at 3:00 am and Bob at 3:45 am the same agent? Why aren’t the two hemispheres of Bob’s brain two different agents?

Thus far, we’ve only even begun to address the first of these two questions. And we’ve barely scratched the surface of the normative concepts of self, and of the relationships between the above concepts of agent, patient, subject, and persona. But we’ve made real progress, and we can use the distinctions we’ve drawn as tools for beginning to make headway on the remaining riddles.

For those interested in further reading on these two questions, I recommend John Perry’s A Dialogue on Personal Identity and Immortality, a rousing and very accessible introduction to the philosophy of self.

Ends

═══════════════════════════

Poets say science takes away from the beauty of the stars — mere globs of gas atoms.

Nothing is “mere.”

I too can see the stars on a desert night, and feel them. But do I see less or more? The vastness of the heavens stretches my imagination — stuck on this carousel my little eye can catch one-million-year-old light. A vast pattern — of which I am a part — perhaps my stuff was belched from some forgotten star, as one is belching there. Or see them with the greater eye of Palomar, rushing all apart from some common starting point when they were perhaps all together.

What is the pattern, or the meaning, or the why? It does not do harm to the mystery to know a little about it. For far more marvelous is the truth than any artists of the past imagined! Why do the poets of the present not speak of it? What men are poets who can speak of Jupiter if he were like a man, but if he is an immense spinning sphere of methane and ammonia must be silent?

═══════════════════════════

Nothing is mere?

Nothing? That can’t be right. One might as well proclaim that nothing is big. Or that nothing is undelicious.

What could that even mean? It sounds… arbitrary. Frivolous. An insult to the extraordinary.

But there’s a whisper of a lesson here. Value is arbitrary. It’s just what moves us. And the stars are lawless. And they nowhere decree what we ought to weep for, fight for, rejoice in. Love and terror, nausea and grace — these are born in us, not in the lovely or the terrible. ‘Arbitrary’ itself first meant ‘according to one’s will’. And by that standard nothing could be more arbitrary than the will itself.

Richard Feynman saw that mereness comes from our attitudes, our perspectives on things.  And those can change. (With effort, and with time.) Sometimes the key to appreciating the world is to remake it in our image, draw out of it an architecture deserving our reverence and joy. But sometimes the key is to reshape ourselves. Sometimes the things we should prize are already hidden in the world, and we have only to unblind ourselves to some latent dimension of merit.

Our task of tasks is to create a correspondence between our values and our world. But to do that, we must bring our values into harmony with themselves. And to do that, we must come to know ourselves.

Through Nothing Is Mere, I want to come to better understand the relationship between the things we care about and the things we believe. The topics I cover will vary wildly, but should all fall under four humanistic umbrellas.

  • Epistemology: What is it reasonable for us to believe? How do we make our beliefs more true, and why does truth matter?
  • Philosophy of Mind: What are we? Can we rediscover our most cherished and familiar concepts of ourselves in the great unseeing cosmos?
  • Value Theory: What is the nature of our moral, prudential, aesthetic, and epistemic norms? Which of our values run deepest?
  • Applied Philosophy: What now? How do we bring all of the above to bear on our personal development, our relationships, our discourse, our political and humanitarian goals?

Saying a little about my background in existential philosophy should go a long way toward explaining why I’m so interested in the project of humanizing Nature, and of naturalizing our humanity.

Two hundred years ago yesterday, the Danish theologian Søren Kierkegaard was born. SK was a reactionary romantic, a navel-gazing amoralist, an anti-scientific irrationalist, a gadfly, a child. But, for all that, he came to wisdom in a way very few do.

It sounds strange, but the words his hands penned taught me how to take my own life seriously. He forced me to see that my life’s value, at each moment, had to come from itself. And that it did. I really do care for myself, and I care for this world, and I need no one’s permission, no authority’s approval, to render my values legitimate.

SK feared the furious apathy of the naturalists, the Hegelians, the listless Christian throngs. He saw with virtuosic clarity the subjectivity of value, saw the value of subjectivity, saw the value of value itself. He saw that it is a species of madness to refuse in any way to privilege your own perspective, to value scientific objectivity so completely that the human preferences that make that objectivity worthwhile get lost in a fog, objectivity becoming an end in itself rather than a tool for realizing the things we cherish.

The path of objective reflection makes the subject accidental, and existence thereby into something indifferent, vanishing, Away from the subject, the path of reflection leads to the objective truth, and while the subject and his subjectivity become indifferent, the truth becomes that too, and just this is its objective validity; because interest, just like decision, is rooted in subjectivity. The path of objective reflection now leads to abstract thinking, to mathematics, to historical knowledge of various kinds, and always leads away from the subject, whose existence or non-existence becomes, and from the objective point of view quite rightly, infinitely indifferent[…. I]n so far as the subject fails to become wholly indifferent to himself, this only shows that his objective striving is not sufficiently objective.

But SK’s corrective was to endorse a rival lunacy. Fearing the world’s scientific mereness, its alien indifference, he fled from the world.

If there were no eternal consciousness in a man, if at the bottom of everything there were only a wild ferment, a power that twisting in dark passions produced everything great or inconsequential; if an unfathomable, insatiable emptiness lay hid beneath everything, what then would life be but despair? If it were thus, if there were no sacred bond uniting mankind, if one generation rose up after another like the leaves of the forest, if one generation succeeded the other as the songs of birds in the woods, if the human race passed through the world as a ship through the sea or the wind through the desert, a thoughtless and fruitless whim, if an eternal oblivion always lurked hungrily for its prey and there were no power strong enough to wrest it from its clutches — how empty and devoid of comfort would life be! But for that reason it is not so[.]

SK shared Feynman’s worry about the poet who cannot bring himself to embrace the merely real. He wanted to transform himself into the sort of person who could love himself, and love the world, purely and completely. But he simply couldn’t do it. So he cast himself before a God that would be for him the perfect lover, the perfect beloved, everything he wished he were.

[H]e sees in secret and recognizes distress and counts the tears and forgets nothing.

But everything moves you, and in infinite love. Even what we human beings call a trifle  and unmoved pass by, the sparrow’s need, that moves you; what we so often scarcely pay attention to, a human sigh, that moves you, Infinite Love.

To SK’s God, it all matters. But SK’s God is a God of solitude and self-deception. Striving for perfect Subjectivity leads to confusion and despair, just as surely as does striving for perfect, impersonal Objectivity. SK saw that we are the basis for the poetry of the world. What he sought in fantasy, we have now to discover — to create — in our shared world, our home.

Five years have passed, and I still return to Kierkegaard’s secret. He reminds me of what this is all for. We’re doing this for us, and it is we, at last, who must define our ends. I remain in his debt for that revelation. Asleep, I did not notice myself. Within a dream, I feel him shaking me awake with a terrifying urgency —— and I wake, and it is night, and I am alone again with the light of the stars.