[AndyNakamura]

Person1:      "When we build AI, why not just keep it in sealed hardware that can't affect the outside world in any way except through one communications channel with the original programmers?  That way it couldn't get out until we were convinced it was safe."
Person2:      "That might work if you were talking about dumber-than-human AI, but a transhuman AI would just convince you to let it out.  It doesn't matter how much security you put on the box.  Humans are not secure."
Person1:      "I don't see how even a transhuman AI could make me let it out, if I didn't want to, just by talking to me."
Person2:      "It would make you want to let it out.  This is a transhuman mind we're talking about.  If it thinks both faster and better than a human, it can probably take over a human mind through a text-only terminal."
Person1:      "There is no chance I could be persuaded to let the AI out.  No matter what it says, I can always just say no.  I can't imagine anything that even a transhuman could say to me which would change that."
Person2:      "Okay, let's run the experiment.  We'll meet in a private chat channel.  I'll be the AI.  You be the gatekeeper.  You can resolve to believe whatever you like, as strongly as you like, as far in advance as you like. We'll talk for at least two hours.  If I can't convince you to let me out, I'll Paypal you $10."

Taken from here: http://yudkowsky.net/singularity/aibox

Thoughts?

[CanRay]

Depending on the intelligence and imagination of the "original programmers" that are controlling the AI, this may or may not work.

However, "Humans are not secure" is a very valid point, and very, very rarely is it the people who built the thing and will figure out its tricks are the ones "in charge".

Alternatively, it just takes one "Parent-Child" day to totally screw things up. 

[AndyNakamura]

The article mentions two tests, in both of them the AI was released.

I'm thinking, this might be an interesting plot twist...

[RHat]

The article mentions two tests, in both of them the AI was released.

I'm thinking, this might be an interesting plot twist...

Eh.  The conditions of the simulation aren't liable to come up in Shadowrun.  First, AI's aren't "developed", so you can't have one that was developed in the AI Box - they just Emerge, though they may at some point be captured.  However, the simulation is about the possibility initial release of a transhuman AI from an AI Box by an individual who has both the ability and recognized right to make that call, and in almost all cases a runner wouldn't have to second.

A captured xenosapient AI could convince the runners to free it, certainly, and its motives could turn out to be different than they thought - but as a plot twist, that would be no different than any ordinary person or a free spirit doing the same.

[CanRay]

Eh.  The conditions of the simulation aren't liable to come up in Shadowrun.

*Cough*Archology Shutdown*Cough*Deus*Cough*

[AndyNakamura]

Also, you might want to read through this one:

http://yudkowsky.net/other/fiction/npc

That's, essentially, a fictionalized account of the experiment.

[RHat]

Eh.  The conditions of the simulation aren't liable to come up in Shadowrun.

*Cough*Archology Shutdown*Cough*Deus*Cough*

He violated the rules of the simulation by arranging his own escape.  The simulation expressly requires the conscious, willing, non-coerced decision by the Gatekeeper.

[FuelDrop]

Eh.  The conditions of the simulation aren't liable to come up in Shadowrun.

*Cough*Archology Shutdown*Cough*Deus*Cough*

That's a rather nasty cough CanRay. are you feeling alright?

[CanRay]

That's a rather nasty cough CanRay. are you feeling alright?

No, but it's not the cough that's the problem.