When to pull the trigger

A game-theoretic view of the singularity

Seth Baum recently wrote an excellent piece about when to launch a Friendly AI. To my mind, there has been surprisingly little discussion of this issue, which seems to me to be fundamental to anyone who's seriously working on a seed AI. While I believe that Seth's analysis is essentially correct, I'd like to lay out my own perspective, which is slightly different and perhaps slightly more concrete.

The basic issue is this: assume that you've developed a seed AI, which when activated is intended to launch a beneficial singularity. However, you are not 100% certain that you've got it right, and you know that activating a flawed AI could cause a catastrophe of almost unimaginable proportions. Should you go ahead and activate your AI, or should you instead wait and continue to work on improving it?

The naive answer (which I believe is more or less the SIAI party line) is that it's immoral to activate any seed AI whose Friendliness is not absolutely certain. The problem is that you aren't working in a vacuum: many other people are working on AIs of their own, and even though you may be the first to develop a viable AI, you certainly won't be the last. If you wait too long before activating your AI, somebody else will beat you to it. Many of the other AI developers are as careful and moral as you are, but some are not. Some of them are well-meaning but reckless, while some are actively malicious (or at least profoundly misguided).

The issue, then, isn't as simple as waiting until you're certain that you've got it right. Instead, your goal should be to maximize the expected Friendliness of the first seed AI to be activated. In some cases, that may mean postponing the launch of your own AI while you continue to improve it. In other cases, it may mean that the most prudent thing to do is to pre-empt the launch of an especially dangerous AI by activating your own AI even though you aren't completely certain that it's Friendly.

This line of reasoning has some obvious consequences at the moment when you're deciding whether or not to activate your AI. Although you're working with very imperfect data regarding the existence, imminence, and probable Friendliness of competing projects, the basic decision process is straightforward. Rather than hashing through the obvious details of what to do at the moment of potential activation, I want to focus on what you should do much earlier on. In particular, I think that the game-theoretic perspective suggests four important points. I'll discuss these points in greater depth in future posts, but for now I want to lay out some basic considerations.

Firstly, it's important to realize that at the end of the day, you should view other well-intentioned AI developers as collaborators rather than competitors. What matters (from a selfish perspective as well as an altruistic one) is not whether your AI is the one that launches the singularity, but rather that the singularity is beneficial rather than cataclysmic.

Secondly, part of the dilemma comes from the fact that activating a traditional seed AI is an all-or-nothing proposition. Once a self-improving super-intelligent conscious being is set loose, conventional wisdom (which I agree with) holds that there's no way to get it back into the box. To my mind, the game-theoretic perspective merely highlights the desirability of developing an AI which has a smooth, controllable takeoff. Friendliness is one way of achieving that goal, although I believe there are better approaches.

Thirdly, time to completion is an important consideration for any serious seed AI project. A fabulous, infinitely Friendly AI that takes a decade longer to develop than its competitors is likely to be irrelevant, no matter how good it is on paper.

Fourthly, information is critical. Obviously, you need accurate information about your competitors in order to make a good launch decision. Equally importantly, they need accurate information about your project. Remember, it's in your best interest that your competitors make optimal decisions about activating their AIs.

As ethical AI developers, I believe that each of us has an obligation to seriously consider and act on each of these points. In addition, it seems to me that there's a role for organizations like SIAI to play in helping to ensure that individual developers make good decisions.